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.ABSTRACT 


A  raxrber  of  problems  that  arise  in  radar  and  sonar  applications 
can  be  regarded  as  parameter  estimation  problems,  in  which  the  desired 
signal,  f(t,a),  is  imbedded  in  non-vciite,  Gaussian  noise.  It  is  de¬ 
sired  to  estimate  the  unknown,  nonrandan  parameter  vector,  o,  from  ob¬ 
servations  (continuous  or  sampled)  of  the  received  noisy  signal  over  a 
finite  time  interval  [0,T] .  Here  f(t,a)  is  a  known  nonstochastic 
function,  and  we  shall  consider  the  case  when  f(t,a)  is  linear  in  a. 

In  this  case,  a  is  referred  to  as  a  linear  regression  vector. 

We  shall  investigate  the*  variance  of  the  Least-Squares  (LS)  esti¬ 
mator  and  of  the  so-called  Generalized-Least-Squares  (GLS)  estimator 
for  a.  Both  are  unbiased  estimators  for  a. 

When  the  noise  covariance  function  is  completely  known  one  may 
construct  a  minimum  variance  unbiased  estimator  (MVUE)  for  a,  and  this 
estimator  is  a  member  of  the  class  of  GLS  estimators. 

Our  interest  is  in  the  case  when  the  noise  covariance  is  not  com¬ 
pletely  known,  but  may  be  regarded  as  a  known  function  of  a  finite 
number  of  unknown,  nonrandom  parameters,  J3. 

It  is  shown  that  when  B_  contains  any  covariance  parameters  other 
than  the  noise  variance,  there  exists  no  MVUE  for  a. 
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However,  we  shall  exhibit  a  class  of  problems  for  which  the  MViE 
for  a  has  a  variance  which  is  orders  of  magnitude  smaller  than  that  of 
the  IS  estimator.  In  such  a  case  it  is  of  interest  to  find  an  estima¬ 
tor  which  makes  use  of  whatever  covariance  information  is  available  in 
an  attempt  to  approach  the  performance  of  the  MViE. 

It  is  shown  that  we  can  significantly  improve  upon  the  IS  estima¬ 
tor  by  employing  a  bootstrapping  procedure  to  estimate  a.  In  some 
cases  the  bootstrapped  estimate  of  a  can  be  shown  to  be  unbiased.  In 
any  case.  It  is  demonstrated  via  computer  simulation  that  the  boot- 
strapped  estimate  of  a  is  capable  of  reducing  the  variance  of  the  IS 
estimate  by  orders  of  magnitude.  In  fact,  the  mean  squared  estimation 
error  using  the  bootstrapped  estimator  for  a  may  be  within  a  few  per¬ 
cent  of  the  variance  of  the  MVUE,  i.e.,  the  variance  the  MVUE  would 
have  if  6_  were  known  a-priori, 

Ihe  bootstrapping  procedure  consists  of  using  the  LS  estimate  of 
a  to  provide  an  initial  estimate  of  the  regression  vector  from  which 
an  initial  estimate  of  the  unknown  covariance  parameters  is  con¬ 
structed. 

IVto  procedures  are  outlined  to  accomplish  the  estimation  of  B_. 

Ihe  first  approach  is  based  upon  an  application  of  the  theory  of 
locally  best  unbiased  estimation.  Ihe  second  approach  is  herein  termed 
the  "inverse-covariance-function"  technique.  Because  of  its  simplic¬ 
ity,  the  latter  approach  is  employed  in  the  simulations. 
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Regardless  of  the  Earner  in  which  the  covariance  parameters  are 
estimated,  these  estimates  are  used  to  construct  the  GLS  estimator  for 
a.  This  is  the  first  iteration  of  the  bootstrapping  procedure. 

The  GLS  estimate  of  is  then  used  to  re-estimate  the  unknown  co- 
variance  parameters,  and  then  to  re-estimate  the  regression  parameters. 

The  process  uses  only  the  one  available  record  of  data,  and  may 
be  repeated  ad  nauseam.  However,  dramatic  results  were  obtained  after 
only  two  iterations  of  the  bootstrapping  procedure . 
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Chapter  1 

mraroucEicN 

1.1  Problems  an!  Objectives 

A  rsisber  of  problems  that  arise  in  radar  and  scalar  applications 
can  be  regarded  as  parameter  estimation  problems  in  viiich  the  data 

r(t)=f(t,o)  +  e(t)  (1-1) 

are  received  (either  continuously  or  sampled)  over  a  finite  time  in¬ 
terval  [0,T] .  Here  the  desired  signal,  f  (t,a)  is  a  known  function  of 
the  unknown  n-dimensional  parameter  vector,  a,  and  e(t)  is  observation 
noise.  We  shall  be  interested  in  the  case  where  a  is  a  nonrandom 
parameter  vector  and  e(t)  is  a  sample  function  from  a  contimous-in- 
mean,  zero  mean,  Gaussian  random  process.  We  shall  restrict  the  dis¬ 
cussion  to  linear  parametric  dependence.  That  is, 

f(t,a)  =  l  «*g*(t)  '  (1-2) 

1=1  1  1 

where  (gi(t)  :i=l,...,n}  are  known  nonstochastic  time  functions. 

It  is  well  known  that  with  the  above  assumptions,  the  minimum 
variance  unbiased  estimator  (MVUE)  of  the  regression  parameter  vector, 
a,  is  a  function  of  the  covariance  properties  of  the  noise  process. 

In  this  dissertation,  we  consider  the  problem  of  estimating  a 
when  the  noise  covariance  (or  equivalently  the  spectrum  in  the  case  of 
a  stationary  random  process)  is  not  completely  known. 
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The  method  of  approach  is  to  treat-  the  noise  covariance  function 
as  a  known  function  of  a  finite  number  of  unknown,  nonrandom  para¬ 
meters. 

The  main  objectives  of  this  work  are: 

1)  to  provide  some  insight  into  cases  where  dramatic 
improvement  over  the  performance  of  the  single  least- 

4* 

squares  (IS)  estimate  ,  which  makes  use  of  no  knowledge 
of  the  noise  covariance,  is  theoretically  possible. 

2)  to  propose  a  reasonable  estimation  procedure  which 
is  capable  of  offering  significant  improvement  over 
the  performance  of  the  IS  estimator. 

3)  to  investigate  the  performance  of  this  procedure 
analytically  and  experimentally  via  several  examples 
which  demonstrate  its  utility. 

1.2  Sutmary  of  Previous  Work 

The  subject  of  optimum*  estimation  of  f(t,a)  when  the  noise 
statistics  are  conpletely  known  has  been  investigated  by  many  authors . 
Rao’s  book  [1]  is  an  excellent  reference  on  the  subject.  Grenander 
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See  Section  2.2  for  the  definition  of  the  LS  estimate. 


In.  the  sequel  optimum  always  refers  to  minimum  mean  squared  error. 
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and  Rosenblatt  [2]  present  the  material  more  from  the  point  of  view 
required  in  the  treatment  of  randcm  processes  than  does  Rao.  Parzen 
[3j  *1]  uses  the  tools  of  Hilbert  space  to  provide  an  approach  which 
is  applicable  to  both  the  continuous-time  and  sampled  data  cases. 
Cramer  [5]  and  Wilks  [6]  are  good  references  for  an  understanding  of 
the  basic  mathematical  statistics  required  in  the  study  of  parameter 
estimation  problems.  Swerllng  [73  derives  useful  expressions  for 
evaluating  the  covariance  matrix  of  vector  parameter  estimates  far  a 
wide  class  of  problems.  In  another  paper  [8]  he  discusses  various 
approaches  to  the  parameter  estimation  problem,  including  the  case  in 
which  the  unknown  parameters  are  regarded  as  having  a  known  a-priori 
probability  distribution. 

On  the  subject  of  power  spectmm  and  covariance  function  estima¬ 
tion  the  classical  references  are  Blackman  and  Tukey  [9]  and  Grenander 
and  Rosenblatt  [2] .  A  recent  paper  by  Parzen  [10]  summarizes  much  of 
the  present  state  of  knowledge  on  this  subject.  All  of  these  works 
are  concerned  with  estimation  of  the  entire  structure  of  the  spectrum 
or  covariance  function,  however,  and  are  not  applicable  to  the  case 
in  which  the  covariance  or  spectrum  is  known  but  for  a  finite  number 
of  unknown  parameters. 

Two  papers  concerned  with  the  parametric  approach  to  power  spec¬ 
trum  estimation  are  those  by  Levin  [11]  and  Hofstetter  [12].  Levin’s 
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results  are  approximate  results  which  hold  under  a  wide  variety  of 
conditions.  Unfortunately ,  however,  the  effects  of  the  approximation 
have  not  been  quantitatively  evaluated.  Furthermore,  for  the  examples 
presented  in  this  dissertation,  where  the  optimum  estimator  is  signi¬ 
ficantly  better  than  the  LS  estimator,  it  can  be  shown  that  Levin's 
approximations  are  invalid.  And  while  Hofstetter's  approach  is  exact, 
he  is  unable  to  obtain  any  analytic  results  except  for  the  case  of 
estimating  a  spectral  amplitude  scale  factor.  The  estimation  of  a 
spectral  scale  factor  is  not  pertinent  to  the  problem  of  estimating 
linear  regression  parameters,  which  we  are  concerned  with  here.  . 

A  recent  paper  by  Rao  [13]  considers  the  linear  regression 
parameter  estimation  problem  when  the  noise  covariance  is  unknown,  and 
an  estimate  of  it  is  incorporated  into  the  regression  parameter  esti¬ 
mate.  Hera  the  noise  covariance  is  regarded  as  completely  unknown, 
and  estimates  are  constructed  for  each  element  of  the  noise  covariance 
matrix.  Rao's  assumptions  require,  however,  that  a  number  of  in¬ 
dependent  realizations  of  the  same  random  process  be  simultaneously 
obtained  (e.g.,  via  multiple  sensors).  (These  separate  realizations 
could  be  correlated  if  their  correlation  is  known  completely . )  In 
fact,  the  number  of  realizations  obtained  must  be  at  least  as  great  as 
the  number  of  time  samples  available  in  each  realization.  This 
follows  from  the  conditions  for  the  existence  of  the  Wishart  distribu¬ 
tion  which  is  central  to  his  results.  In  his  paper,  Rao  uses  the 


5 


available  independent  realizations  in  order  to  construct  the  covar¬ 
iance  estimates. 

Up  to  this  point,  references  have  been  cited  which  fall  into 
one  of  three  categories: 

1)  Signal  parameter  estimation  with  known  noise  covariance 

2)  Noise  spectrum  or  covariance  estimation  (parametric  and 
nonparametric) 

3)  Regression  parameter  estimation  using  an  estimated  (non¬ 
parametric)  noise  covariance  matrix. 

The  parametric  covariance  approach  to  be  adopted  here  really 
amounts  to  a  particular  kind  of  joint  parameter  estimation  problem. 

As  such,  it  is  logical  to  ask  if  there  exists  seme  general  theory 
which  is  capable  of  providing  optimum  estimators  of  the  desired  para¬ 
meters. 

Ihe  results  of  Barankin  [14]  provide  the  desired  theory  when 
one  is  willing  to  accept  what  are  termed  "locally  best"  unbiased 
estimates  of  the  desired  parameters.^  In  some  cases  only  locally 

^We  are  not  really  interested  in  locally  best  estimates  of  cu  How¬ 
ever,  an  application  of  Barankin* s  theory  will  reveal  that  under 
certain  conditions  (to  be  stated)  no  MVUE  of  on  exists.  We  will  then 
abandon  the  requirement  of  bestness  and  hope  to  find  an  estimate  for 
£  whose  performance  is  close  to  that  of  the  MVUE  when  the  latter 
exists. 
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best  unbiased  estimates  will  exist .  A  necessary  and  sufficient  con¬ 
dition  for  the  existence  of  locally  best  and  for  uniformly  best 
(i.e.,  MVUE)  unbiased  estimates  was  stated  by  Rao  [1].  Barankin  gives 
a  different  necessary  and  sufficient  condition  for  the  existence  of  a 
locally  best  unbiased  estimate.  Swerling  [15]  restates  Barankin's 
results  in  a  form  more  easily  applied.  Since  neither  of  these  papers 
has  received  much  attention,  and  because  of  the  author's  own  interest 
in  this  important  subject,  much  of  the  development  will  be  elaborated 
upon  in  this  dissertation. 

In  addition  to  the  above-mentioned  references ,  there  are  a  host  of 
papers  [16-33]  in  the  fields  of  cybernetics  and  adaptive  control  which 
the  author  has  found  useful  and  stimulating. 

1.3  Outline  of  Dissertation  and  Sunmary  of  New  Results 

Chapter  II  begins  with  an  investigation  of  the  simple  LS  estima¬ 
tion  procedure  and  proceeds  through  a  development  of  the  generalized- 
least-squares  (GLS)  estimator*.  The  optimum  estimator,  when  the  noise 
covariance  is  known,  is  a  memoer  of  the  class  of  GLS  estimators.  For- 


Ihe  GLS  estimator  is  defined  in  Section  2.3. 
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riuilas  are  given  for  the  covariance  matrix  of  the  parameter  estimates.. 
Hie  results  are  given  for  the  continuous-time  and  sampled  data  cases. 
These  results  are  not  new,  but  their  development  constitutes  a  con¬ 
venient  introduction  to  the  subject  and  symbology  which  follow  them. 
Using  these  tools,  two  examples  are  provided  in  which  dramatic  im¬ 
provement  over  the  performance  of  the  IS  estimator  is  theoretically 
possible;  These  examples  are  investigated  in  greater  depth  in  the 
chapters  which  follow. 

In  Chapter  III  the  Cramer-Rao  bounds  for  joint  unbiased  estimates 
of  the  regression  and  covariance  parameters  are  derlvedi  for  the  first 
example  mentioned  above.  In  this  exanple  the  noise  process  is  a 
stationary  first-order  autoregressive  scheme-.  It  is  shown  that  the 
bound  on  the  regression  parameter  estimate  is  not  increased  by  the 
-presence  of  the  unknown  noise  covariance  parameters.  Since  this  bound 
cannot  be  achieved  except  under  certain  limiting  conditions,  an  in¬ 
vestigation  into  the  Barankin  bound,  which  is  always  achievable,  is 
made.  The  Barankin  bound  analysis  is  quite  general  and  is  not  re¬ 
stricted  to  the  case  of  autoregressive  noise.  (An  exposition  of 
Barankin’ s  theory  is  given  in  Appendix  II.)  It  is  shown  that  no  uni¬ 
formly  best  (i.e.,  MVUE)  estimator  for  the  regression  parameter  exists 
when  the  noise  covariance  (normalized)  has  unknown  parameters.  A 
technique  employing  barankin' s  theory  for  the  estimation  of  pertinent 
covariance  parameters  is  suggested. 


Chapter  IV  is  concerned  with  the  synthesis  and  analysis  of  the 
desired  estimation  procedure.  The  procedure  involves  nonlinear  opera- 
tions,  on  the  data,  making  analytic  results  for  bias  and  mean  square 
error  calculations  difficult  to  obtain.  Sane  analytic  results  regard¬ 
ing  the  bias  of  the  estimator  are  presented.  A  detailed  investigation 
into  the  performance  of  the  procedure  is  made  via  a  digital  computer 
simulation  of  the  two.exanples  mentioned  earlier.  In  the  course  of 
developing  the  proposed  procedure,  a  comparison  is  made  to  the  joint 
maximum  likelihood'  estimator. 

The  main  result  is  the  demonstration,  (via  simulation)  that  signi¬ 
ficant  Improvement  over  the  IS  estimator  performance  can  be  obtained 
in  same  cases, by  the  use  of  the  proposed  procedure.  The  procedure 
actually,  amounts  to  an  unsupervised  learning  or  "bootstrapping"  tech¬ 
nique  in  which  the  IS  estimate  of  the  regression  parameter  is  used  to 
estimate  the  noise  covariance  parameters .  The  estimated  covariance 
parameters-  are  then  used  to  revise  the  regression  estimate  making  use 
of  the  GLS  estimator.  The  process  can  be  continued  ad-nauseam . 
However,  the  exanples  studied  indicate  that  after  only  two  revisions 
of  the  IS  regression  estimate,  the  mean  squared  error  may  be  nearly 

o 

equal  to  the  variance  of  the  MVUE. 

This  result  runs  counter  to  a  remark  made  by  Eicker  [3^]  that  a 
useful  estimate  of  the  covariance  matrix,  or  of  the  functions  of  it 
which  are  pertinent  to  estimating  the  regression  parameters,  cannot  be 
constructed  from  a  single  finite  sequence  of  observations. 


Chapter  2. 
LINEAR  ESTIMATION 


2.1  Introduction 

By  a  linear  estimator,  a,  is  meant  a  measurable  function  of  the 
observables,  R,  which  is  linear  in  the  observables.  The  interest  in 
linear  estimators  is  due  largely  to  the  ease  with  which  they  can  be 
synthesized  and  also  to  the  availability  of  analytic  tools  to  investi¬ 
gate  their  performance.  Furthermore,  for  the  linear  regression  model 
of  interest  here,  the  MVUE  of  a  is  a  linear  estimator. 

In  this  chapter  the  LS  and  GLS  estimators  of  ct,  which  are  linear, 
unbiased  estimators  of  a,  will  be  investigated.  The  data  will  be  as¬ 
sumed  to  consist  of  time  samples,  and  vector-matrix  notation  will  be 
employed  for  convenience.  The  generalizations  required  for  treatment 
of  the  continubus-time  problem  will  also  be  given.* 


In  general  the  results  for  time  sampled  problems  will  depend  upon 
how  the  time  samples  are  distributed  on  the  observation  interval. 
This  dependence  will  not  be  explicitly  denoted  in  the  sequel, 
however . 

It  is  well  to  observe  that  if  the  time  samples  are  constrained  to 
be  equispaced  on  an  observation  interval  of  fixed  length,  the 
estimation  accuracy  is  not  necessarily  mohotonically  improved  as 
the  number  of  samples  is  increased.  It  can  be  shown,  however, 

[35]  that'  the  optimum  linear  estimate  is  obtained  for  dense  sampling 
in  the  interval.  (Also,  of  course,  a  set  of  samples  which  con¬ 
tains  another  set  will  not  lead  to  a  worse  estimate.) 
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2.2  Least-Squares  Estimation, 

Hie  problem  of  interest  may  now  be  stated  as  the  estimation  of  a 
when  the  data  is  of  the  .form 


(2-6) 


The  system  of  equations  above  is  sumarlzed  by  the  equation 

T  T  a 

g*  =  gg2ls 

which  yields  the  familiar  result* 

=  [GtG']“V R  (2-7) 

It  is  easy  to  see  that  is  an  unbiased  estimate  of  a.  That 
is,  let  a  belong  to  the  parameter  set  A.  Then** 

El^]  =  ay  ae  A  (2-8) 

The  covariance  matrix  of  any  unbiased  estimate  of  a,  say  a, 
(also  termed  the  estimator  dispersion  matrix)  is  an  nxn  matrix  of 
elements, {d^ ;  i,j  =  l,...,n} which  we  will  denote  by 


D(a)  =  [d±_.  ]  =  E[(a  -  a)  (a  -  a)T] 

(2-9) 

For  the  IS  estimate  we  have 

D(oLS)  =  [GTG]_1GT4>  G[GTG]_1 

(2-10: 

where  $  is  the  noise  covariance  matrix, 

*  T  -1 

The  existence  of  [G  G]  is  assumed.  This  requires,  of  course, 
that  N  £  n. 

The  notation  E[...]  denotes  ensemble  expectation  and  will  be  used 
interchangeably  with  the  symbol  [...). 
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♦  -[*_]  «E[  e  eT]  (2-11) 

jiv  - 

Observe  that  is  linear  in  the  data,  and  that  it  in  no  way 
involves  knowledge  of  the  noise  covariance  matrix  for  its  construc¬ 
tion. 

To  derive  the  corresponding  formulas  for  the  continuous-time 
case  we  may  proceed  by  making  use  of  the  Karbunen-Loeve  expansion  of 
the  randan  process  e(t) .  Iben  for  a  positive  definite  covariance 
function,  ♦(t,s) 

♦(t,s)  «.S(e(t)e(s)]  (2-12) 

T 

\^(s)  *  j  f(t,s)^(t).  dt  (2-13)  * 

'  0 

T 

J  ^k(t)ipt(t)  dt  ■  6fcjl  (2-1*1) 

0 

we  have 

T 

rk  =  J"  r(t)ipk(t)  dt  (2-15) 

0 

T 

gik  =  J  g^OO^Ct)  dt  '  (2-16) 

0 

T 

ek  =  /  e(t^k(fc)  dfc  / 

0 


(2-17) 


where  the  above  equations  hold  for  k  -  1,2, . . . . 


Hence,  we  have 
n 

rk  *23  “Ak  +  \  ;  k  *  1»2»***  (2-18) 

1-1 

or  in  vector-matrix  form 

R-Ga  +  e  '  (2-19) 

Equation  (2-19)  is  exactly  the  same  as  Equation  (2-1),  except 
that  in  (2-19)  the  dimensions  of  the  vectors  and  matrices  are  infinite 
with  respect  to  the  k  index,  lhe  solution  for  obtained  in  Equa¬ 
tion  (2-7),  therefore,  still  holds  with  the  following  modifications. 

T 

The  nxn  matrix  G  G  has  elements  given  by 

oo  oo  f 

(gTg)ij  *2  giAj  *2  /  gike»j*k(t>*»(t)  * 

k=l  k,m=l  q 


00 

giA(«)  (  2 


dt 


T 

=  j  g±(t)  gj(t)  dt  j  i,j  =  1,2,..., n  (2-20) 

0 

T 

Similarly,  the  nxl  vector  G  R  has  elements  given  by 
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®  T 

(GTR)j  *  2  gjkrk  “  f  r(t)  dt  ;  j  =  1,2,..., n  (2-21) 

le*l  0 

For  the  estimator  dispersion  matrix.  Equation  (2-10)  still  holds, 
where  (2-20)  is  used  to  obtain  the  matrix  G  G,.  and  where  the  elements 
of  the  matrix  GT$  G  are  found  from 


(G  $  G)^  85  gik  ^km  ”  1,2,..., n 

.k,mFl 


(2-22) 


wfth; 


T 


^km  "  //  $  (t,s)if>k(t)i{>m(s)  dt  ds  =  Ak  6 


km 


(2-23) 


0 


Substituting  (2-23)  into  (2-22)  we  obtain  the  desired  expression. 
T 

(GT$  G)i:j  =  //  gi(t)'  4>  (t,s)  gj(s)  dt  ds  ;  i,j=l,2,...,n  (2-24) 

J  0 

2.3  Generalized  Least  Squares  Estimation 

In  the  preceding  section,  the  LS  estimator  for  a  was  found  by 
minimizing  the  sum  of  the  squares  of  the  residuals.  The  GLS  estimator 
is  obtained  by  minimizing  the  more  general  quadratic  form 


Q 


N  .  n  x  .  n  . 

<X&~  22  K  "  2  VW  |  rv  -  E  “iSivl^v 

Vi  iV=l  ^  i=l  '  *  i=l  1 


(2-25) 
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where  nyv  are  the  elements  of  an  arbitrary  (real)  positive-definite 
matrix,  n.  Usually,  n  is  taken  to  be  symmetric.  It  is  apparent  that 
when  n  is  chosen  to  be  the  identity  matrix,  the  LS  estimator  will 
result.  It  will  be  seen  shortly  that  2gls  811  unbiased  estimator  of 
a  and  is  linear  in  the  data. 


For  Gaussian  noise  with  covariance  matrix  4>,  if  n  is  taken  to  be 
$_1,  then  cLTC  is  the  MVUE  of  a.  Ms  can  be  shown  in  a  variety  of 

“vLS 

ways  [1,  2]. 

For  the  present  we  shall  regard  n  as  any  real,  synmetric,  pos¬ 
itive-definite  matrix  and  give  the  form  of  o^g. 

Proceeding  formally  with  differentiation  as  in  Equation  (2-5)  the 


following  matrix  equation  results . 


[GTnG]  =  GTnR 


Hence, 


a  T  -1  T 

o-j.g  =  (GnG]  GnR 


(2-26) 


(2-27) 


Observe  that  a^g  is  linear  in  R  and  is  an  unbiased  estimate  of  a,  no 
matter  what  matrix,  n>,  is  used.  Ihe  dispersion  matrix  of  is 


easily  calculated'. 

D(oLT<I)  =  [GTr|G]  -1GTn$nG  [GTnG]_1 


(2-28) 


In  the  case  where  r\  =  $  this  reduces  to  the  well-known  result 
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D(a^)  »  [gV^G]"1  (2-29) 

where  the  subscript  cn  ^  denotes  the  fact  that  this  is  the  MVUE  of 
o.  Note  from  (2-27),  with  n  -  $-1,  the  explicit  dependence  of  ^  on 

Q 

the  (normalized),  noise  covariance  matrix. 

The  formulas  for  the  continuous  time  case,  corresponding  to 
Equations  (2-20),  (2-21)  and  (2-2*0,  for  arbitrary  n  require  special 
treatment .  When  the  sanpled-data  form  of  n  is  the  inverse  of  a 
covariance  matrix  resulting  from  the  sarrpled  form  of  a  covariance 

A 

function,  <J>(t,t# ),  the  results  of  Swerling  [7]  may  be  used  to  find 
the  formulas  in  the  limit  as  the  sanples  become  dense  in  the  observe- 
tion  interval.  We  will  be  interested  in  the  dispersion  matrix  of 

A 

when  the  sampling  is  dense.  In  this  case 
T 

(gVS)^  =  J  g1(t)  h  (t)  dt  j  i,j  =  1,2,... „n  (2-30) 

0 

where  (t)  is  the  solution  to  the  integral  equation 
T 

Jl-v(t)  4>(t ,s)  dt.  =  g  (s)  ;  s  e  [0,T]  ;  j  =  1,2,..., n  (2-31) 

0 

This  result  is  valid  whenever  {h^(t)  ;  j  =  1,2,..., n|  exist  as  a 
solution  to  the  above  integral  equation  and  whenever  certain  other 
conditions  outlined  in  Swerling  (t)  and  [15]  are  satisfied. 

A 

The  estimator  dispersion  matrix  for  a  is  then  found  using 
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Equation  (2-29). 

With  the  preceding,  results  at  our  disposal,  we  are  prepared  to 
compare  the  performance  of  the  LS  estimator  with  that  of  the  MVIE. 

2.4  Performance  Ccnparison 

The  purpose  of  this  section  is  to  disclose  a  class  of  problems 
in  which  "dramatic  Improvement"  over  the  performance  of  the  IS  esti¬ 
mator  would  be  possible  if  the  noise  covariance  were  known.  By 
dramatic  here,  we  mean  at  least  an  order  of  magnitude  reduction  in 
the  mean  square  estimation  error. 

This  investigation  is  motivated  in  two  ways.  Firstly,  the  com¬ 
parison  provides  a  measure  of  the  sensitivity  of  the  optimum  estimator 
to  lnpreeise  knowledge  of  the  noise  covariance.  Highly  sensitive 
cases  will  exhibit  dramatic  performance  differences.  Secondly,  since 
a  simulation  is  enployed  to  investigate  the  performance  of  the  esti¬ 
mation  procedure  suggested  in  Chapter  IV,  it  is  desirable  to  simulate 
examples  where  the  improvement,  if  any,  is  not  likely  to  be  obscured 
by  simulation  innaccuracies .  The  results  of  this  section  provide  such 
examples. 

The  following  treatment  will  be  concerned  with  the  case  when  a 
consists  of  only  a  single  parameter,  a.  The  formulas  presented  in  the 
previous  sections  can  be  used-  for  the  more  general  vector  parameter 
case  in  an  obvious  manner.  Also,  the  continuous-time  formulas  will 
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be  enphasized  since  their  use  avoids  the  need  for  matrix  inversions 
and  facilitates  hand  calculations. 

o  *»  •» 

therefore,  for  the  variance  of  the  IS  estimator  we  have  from 
(2-10),  (2-20)  arid  (2-24) 


T 

:  If  zM  4>(t,s)  g(s)  dt.  ds 

V.  -  »  - - r - —  (2-32) 


and  the  minimum  variance  possible  is 


o 

V 


h(t)  <j>  (t,s)  h(s)  dt  ds 


h(t)  g(t)  dt 


i 


-1 


(2-33) 


where  h(t)  is  the  solution  to 
T 

J  h(t)  <J>  (t,s)  dt  =  g(s)  ;  se[0,T]  (2-34) 

0 

o 

We  seek  exanples  for  which  V  is  much  less  than  V^.  One  such 
exanple  is  the  -following: 


Example  1. 

g(t)  =(cos  wot  ;  te[0,T]  (2-35) 

1 0  otherwise 

<t>  ^(t_s)  -  a2  exp[-3  | t—s ( ] *  (2-36) 


Note  that  the  noise  process  is  stationary  in  this  example.  This  will 
also  be  the  case  for  Example  2,  which  follows . 


L9 


where  the  parameters  to  and  6  are  chosen  in  accordance  with  the  con- 

O 

diticn  that  for  sane  non-negative  integer,  k. 


0<3  T«l<uo  T  =  (2-37) 

where  T  Is  the  length  of  the  observation  interval.  If  this  is  done, 
then  for  dense  sampling  we  obtain 
ft  „  (<o  T)2 

v  =  (B  T)  - o - 2  6  T  (2-38) 

LS  (w  Tr  +  23  T 

To  derive  the  above  results  we  apply  Equations  (2-32)  through  (2-3*0. 


a2  JJ  (cosa>ot)  (cosuqs)  exp(-3  |t-s|)  dt  ds 


VLS 


r  }  2  t 

LJ  °  * 


(2-39) 


The  calculation  of  the  integrals  above  results  in 

VTC  =  8a2{(to  T)2[l  +  (3  T/w  T)Z][1  +  (sin  2<o  T)/2to  T]2}-1* 

JjO  '  o  o  oo’ 

|(3  T/2)  [1+  (sin  2<o  T)/2<o  T]  +  (sin2  to  T)/2 
>  op  o 

+[exp(-3  T)  ][(3  T/o)  T)2cosco  T  -  (3  T/to  T)sinto  T]« 

O  O  O  '  o 

[1  +  (3  T/uoT)2]_1  -  (3  T/uT)2[1  +  (3  T/(0oT)2]"1[  (2-40) 

o 

To  calculate  V  we  require  the  solution  for  h(t)  in  Equation  (2-34) 
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Ihis  solution  is  given  in  Swerling  [15]  when  4>  (t,s)  is  of  the  form 
expressed  in  (2-36)  and  g(t)  is  twice  differentiable  with  respect  to 
t  for  t  within. the  observation  interval.  When  g(t)  is  given  by  (2-35) 


the  result  is 


■M 


g  (cosw  t)[l  +  (w  T/B  T)  ]  +  6(t)  + 

O  0 


[cosed  T  -  (w.T)(sinw  T)/B  T]  5(t- 
'O  -O  o 


(2-ill) 


where  <6(.)  is  the  Dirac : delta  function. 

Using  (2-41)  in  (2-33),  we  obtain  for  $ 


(2-42) 


[(wT)2  X  f  <d  T] 

2[tV-  +  6  TJ  +  -H1  +  V1  +  [It  -  ^Jsl"2“oT 

! 

It  can  be  verified  that  the  quantity  ^//VLS  has  local  minima 

when  to  T  is  equal  to  an  odd  multiple  of  tt/2  .  If  u  T  and  3  T  satisfy 
o  •  >  o 

relation  (2-37),  the  approximate  result  in  (2-38)  follows.  Note  that 
the  dramatic  improvement  afforded  by  the  optimum  estimator,  as 
measured  by  depends  on  the  ratio  of  the  signal  bandwidth  to  the 

lib  ' 

noise  bandwidth;  not  on  the  signal-to-noise  power  ratio  per  se. 

While  the  above  analysis  applies  strictly  for  dense  sampling,  it 


provides  a  good  indication  of  the  behavior  of  the  time-sampled  version 
of  the  problem  if  the  sampling  rate  is  sufficiently  high.  Ihis  will 
be  indicated  by  numerical  results  later. 
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Considerable  insight  can  be  gained  by  looking  at  the  above  ex- 
anple  fran  the  frequency  domain  point  of  view.  To  this  end  we  shall 

define  the  signal  power  spectrun,  S  (u>),  as 

s 

s  (u)  ■  l^jg(t)}  |2  =  I  /  g(t)exp(-jut)  dt] 2  (2-43) 


where  *{■■■)  denotes  the  Fourier  transform  operation.  Ihe  noise 
power  spectrum  will  be  defined  in  the  usual  fashion  by 

se(“)  (t-s)}  (2-44) 

From  this  it  follows  that  for  Exanple  1 


sin  x.A  /sin  xA 

~)  +\~) 


cosu>oT[coswoT  -  cos  off] 


X1X2 


j(2-45) 

(2-46) 


where 


(u J  -  u  )  T 
o 


(w  +  a)  )  T 

x  =  _ _ 9 _ 

2  2 


(2-47) 

(2-48) 


Now  it  can  be  observed  that  when  okT  is  fixed  at  an  odd  multiple 

of  tt/2  the  cross-product  term  (i.e_,  the  last  term)  in  S  (w)  vanishes. 

s 

This  permits  some  separation  between  the  main  regions  of  concentration 
of  signal  and  noise  power. 
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Figure  (2-1)  depicts  the  distribution  of  signal  and  noise  power 
in  the  frequency  danain  when  the  parameters  are  chosen  so  that  dra¬ 
matic  inprovement  is  possible.  In  this  illustration  u  T* equals  5ir/2. 

O 

(She  spectra  are  symmetric  about  zero  frequency  and  u  .) 


Figure  2-1.  Power  Spectra  for  Example  1. 

Figure  (2-1)  suggests  that  the  dual  problem,  which  obtains  when 
the  center-frequency  locations  of  the  two  spectra  in  Example  1  are 
interchanged,  might  also  afford  an  exanple  where  dramatic  improvement 
over  the  performance  of  the  LS  estimator  is  possible.  Shis1,  in  fact, 
is  true . 
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2$ 


To  calculate  $  it  is  necessary  to  solve  the  integral  equation 
T 

*?"  :J  h(t)exp(-8  jt-sj)cos[t»c(t-s)]  ct  =  1  ;  se(0,T]  (2-5^) 

o 

for  hft) .  The  details  for  obtaining  this  solution  are  presented  in 
Appendix  I.  The  result  is 

h(t)  *  Xo{exp(-Xt)  +  exp[+k(t-T)]}  +  +  K2£5(t)  +  6(t-T>]  (2-55) 

where 

k2  -  w2  +  32  (2-56) 

C 


Se(u)  *  a" 


r  «cjr 
C3 


{- _ _  +  B  ] 

lB^  +  (u  -  u)^  B^  +  (w  +  to)  j 


(2-62) 

(2-63) 


Figure  2-2.  Power  Spectra  for  Example  2. 

Ihe  above  examples  suggest  that  a  class  of  problems  In  which  $/VTC  is 
very  small  (i.e. ,  «1)  is  characterised  by  the  property  that  the  noise 
bandwidth  is  very  snail  relative  to  the  signal  bandwidth,  and  that  the 
location  of  the  noise  "spike"  is  such  that  a  linear  weighting  filter 
operating  on  the  available  data  can  de-enphasizs  the  noise  energy 
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without  unduly  corrupting  the  useful  signal.  It  would  also  appear 
that  this  concept  could  be  extended  to  noise  spectra  which  contain 
several  spikes. 

In  fact,  these  statements  can  be  rigorously  established  whenever 
the  optimum  linear  unbiased  estimator  for  a  can  be  represented  by 
>  T 

aQ  *  j  #(t)  r(t)  dt  (2-64) 

0 

where  w(t)  denotes  the  optimum  weight  function. 

By  applying  the  variational  calculus  to  minimize  the  variance  of 
T 

a  *  J  w(t)  r(t)  dt  (2-65) 

0 

subject  to  the  unbiasedness  constraint  that 
T 

/  w(t)  g(t)  dt  =  1  (2-66) 

0 

it  is  found  [36]  that  §(t)  must  satisfy  the  integral  equation 
T 

/act)  <j>  (t,s)  dt  =  $  g(s)  ;  s  S[0,T] 

0: 

where  $  is  the  miniimin  variance  obtained,  and  is  given  by 


(2-67) 
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T 

V  =  Jf-  w(t)  4>  (t,s)  w(s)  dt  ds  (2-68) 

0 

(Multiply  both  sides  of  (2-67)  by  8(t)  and  apply  (2-66).) 

But  $  is  also  given  by  Equations  (2-33)  and  (2-3*0,  since  the 
optimum  estimator  is  unique  with  probability  one  (1,:  6],  We  then  have 
the  association  between  w(t)  and  h(t)  given  by 

8(t)  =  $  h(t)-  (2-69) 

Hence,  except  for  a  constant,  the  optimum  weight  function  is  equal  to 
the  solution,  h(t),  of  Equation  (2-3*1). 

Now  for  any  w(t)  in  (2-65)  and  for  staticnaiy  noise,  the  variance 

A 

of  the  corresponding  estimate,  a,  is 
T 

V(a)  =  ff  w(t)  $  (t-s)  w(s)  dt  ds  (2-70) 

0 

And  if  we  define  k(t)  by 

k(t)  =  fw(t)  t  e  [0,T]  (2-71) 

(0  otherwise 

then  making  use  of  the  convolution  theorem  [371  and  Parseval's  formula 
we  have 

00 

V(“)=2?  /  |K(w)|2  Se(u)  dm 

-CO 


(2-72) 


where  K(w)  is  the  Fourier  transform  of  k(t) .  Furthermore,  the  con¬ 
straint  equation  may  be  rewritten  as 


_1 

2rr 


(2-73) 


where  G(oj)  is  the  Fourier  transform  of  g(t). 

We  now  see  that  w(t)  is  such  that  the  integral  in  (2-72)  is 
minimized  subject  to  the  constraint  in  (2-73)-  It  is  in  this  fashion 
that  w(t)  de-errphasizes  the  noise  energy  while  attempting  to  preserve 
the  information  in  the  desired  signal. 

Another  point  of  view  which  is  useful  for  investigating  condi¬ 
tions  for  dramatic  improvement  is  obtained  by  employing  the  Karhunen- 
Loeve  expansion  of  the  noise  process.  This  point  of  view  does  not 


require  that  the  noise  process  be  stationary. 


Suppose,  in  accordance  with  Equations  (2-12)  through  (2-17),  we 


find  the  eigenfunctions  <4  (t)>  and  corresponding  eigenvalues 
t  (  )k=l 

|a.  V  associated  with  the  covariance  function,  4>(t,s).  Then  the 
(  >k=l 

power  in  the  k  noise  expansion  coefficient,  efc,  is  A^,  and  the 
th 

power  in  the  k  signal  expansion  coefficient,  g. ,  is  fk.  Separation 
of  the  signal  and  noise  energy  distribution  occurs  if,  for  every  k, 
g?  A, 

-the  product  ■  T  -  -  is  equal  to  zero.  V/hile  total  separation  cannot 
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normally  be  expected,  it  may  be  sufficient,  in  order  to  obtain 


o 

v/VLS«l,  to  have  -  —  «max  .  T 


2  X,.  !  (g*)1 


,  whenever  either 


(g k)f 

TT-  or 


is  large. 

For  instance.  Van  Trees  [53]  gives  the  eigenfunctions  and 
eigenvalues  for  Example  1  (except  that  the  observation  interval  is 
taken  syirmetric  about  zero  for  convenience).  Hence,  for  Exarrple  1 
with  t  e  [-T/2,  T/2]  we  have 
2 


A  =  .  k  =  l  2 


B  +  b, 


V2 


VF  i 


sin  b.T 

k 

b.T 

k 


-i  \\f2  bkt  *  k 


odd 


(2-74) 


(2-75) 


*k(t)  = 


sin  b,  T 

1/2 

Vt" 

1  -  k  : 
b.T 
k 

sin  bkt  J  k  even 


(2-76) 


when  the  bfc  are  the  solutions  to  the  transcendental  equation 
b. 


1 


tan(bkT/2)  +  ft 


J  L 


tan(bfcT/2) 


6_"|  _ 

"bkJ 


0 


(2-77) 


If  the  values  of  bfc  are  arranged  in  increasing  order  and  u)qT  and 
BT  are  chosen  in  accordance  with  (2-37)  it  is  found  that 

bkT  =  (k  -  1)  £  ;  k  *  4 


(2-78) 
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Meanwhile  we  have 
T/2 

^  =  \  f 

-T/2 


costa  t  cos  b,.t  dt 
o  k 


T\ 


sin[(wQ  -  bfc)T/2]-  sin[(a)Q  +  bk)T/2] 
+  (io  +  b.  )T/2 


'(wo  -  bk)T/2 


,  k  odd  (2-79) 


T/2 

^  =  \  / 

-T/2 

=  -\(T/2) 


cosu  t  sin  b,t  dt, 
o  k 


cos,((bk  -  ojo)T/2]  cos[(bfc  +  u>q)T/2] 


(bk  -  <oo) T/2  +  (bt  +  Uo)T/2 


,k  even  (2-80) 


where 


*k* 


£_ 


i— I  sinb  t! 


T7I  ;  k  054 


V? 


(2-81) 


V_  T  cosb.  T 

T  1 _ 

b.T 

l.  L  k  J 


■j^2  »  k  even 


Using  the  above  results,  it  is  easily  verified  that,  indeed, 

g£Xk  )(sk)2  ( 

,  XkJ  for  all  k. 

The  above  remarks  are  intended  to  serve  as  an  intuitive  guide  to 
a  class  of  problems  where  one  might  expect  to  be  able  to  construct  an 
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estimator  which  is  significantly  better  than  the  LS  estimator.  There 

is,  in  fact,  a  close  analogy  between  the  class  of  problems  mentioned 

above  and  problems  related  to  spatial  filtering  of  directional  noise 

sources  in  radar  and  sonar  applications  [21] .  In  any  case.  Equations 

(2-10)  and  (2-29)  may  be  used  to  investigate  other  specific  examples. 

It  is  well  to  point  out  that  for  large  observation  intervals, 

the  class  of  problems  in  which  the  IS  estimator  can  be  significantly 

improved  upon  is  quite  small  [2,  38] .  Even  Examples  1  and  2  above  are 
o 

such  that  V/V--  approaches  unity  as  'C*®.  On  the  other  hand,  this  in¬ 
dicates  that  many  problems  in  which  D^)  =  D(^g)  for  large  T,  might 
be  more  interesting,  with  regard  to  sensitivity,  when  T  is  small. 


Chapter  3 

BOUNDS  FOR  THE  JOINT  PARAMETER  ESTIMATION  PROBLEM 

3.1  Introduction 

Up  until  now  we  have  been  concerned  with  the  performance  of  the 
LS  estimator  for  a  as  compared  with  that  of  the  MVUE,  which  requires 
ccnplete  knowledge  of  the  normalized  covariance  function  of  the  noise 
for  its  construction.  We  would  like  to  know  the  answer  to  the 
following  question:  What  is  the  limit  on  the  accuracy  of  unbiased  es¬ 
timates  of  a  when  the  normalized  covariance  function,  itself,  has  in- 
known  parameters?  More  precisely,  what  is  the  greatest  lower  bound 
on  the  variance  of  unbiased  estimates  of  a  in  the  presence  Of  unknown 
covariance  parameters?  We  are  also  interested  in  the  form  of  the  es¬ 
timator  which  achieves  this  bound.  This,  then,  is  what  we  will  refer 
to  as  the  joint  parameter  estimation  problem. 

Hence,  consider  the  family  of  Gaussian  probability  density  func¬ 
tions  |p(R;x)  s  Xer}  defined  with  respect  to  Lebesgue  measure  over 
Euclidean  N-space,  E^,  where 

XT  =  [I«]T  (3-D 

denotes  the  (m+n)  -dimensional  parameter  vector  composed  of  the  m-dimen- 
sional  covariance  parameter  vector,  3_,  and  the  n-dimensional  regres¬ 
sion  parameter  vector,  a,  which  belong  to  sane  parallelepiped,  T,  in 

E  .  . 
m+n 
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The  Crainer-Rao  (C-R)  bound  for  the  joint  parameter  estiinaticn 
problem  furnishes  a  lower  bound,  not  only  for  the  variance  of  unbiased 
estimates  of  a,  but  also  for  the  mean  squared  error  of  estimates  for 
the  covariance  parameters.  These  bounds,  however,  are  in  general  not 
the  greatest  lower  bounds,  and  unless  they  are,  they  are  not  attain¬ 
able  by  any  estimator.  Nonetheless,  the  ease  with  which  the  C-R 
bound  can  be  calculated  in  many  cases,  renders  it  an  important  bound. 

We  will  calculate  the  C-R  bound  for  the  joint  parameter  estima¬ 
tion  problem  in  Exanple  1,  assuming  that  unbiased  estimates  for  all 
the  parameters,  exist.  (An  unbiased  estimate  of  the  regression  param¬ 
eter  always  exists;,  namely  the  IS  estimator.  It  can  also  be  shown 

2 

that  an  unbiased  estimate  of  a  exists.  The  investigation  of  whether 
or  not  unbiased  estimates  of  the  parameters  in  the  normalized  covar¬ 
iance  function  exist  when  the  regression  parameter  is  unknown,  is 
beyond  the  scope  of  this  work.  It  is  noted,  however,  that  the  cal¬ 
culations  for  the  C-R  bound  which  follow  can  be  easily  modified  to 
include  the  effect  of  a  known  bias  in  estimating  these  parameters.) 

The  calculation  of  the  C-R  bound  for  Exanple  1  will  reveal  that 

* 

the  C-R-efficient  estimates  of  the  regression  parameters  are  uncor- 

« 

This  terminology  refers  to;  those  estimates,  if  they  exist,  which 
have  the  dispersion  matrix  given  by  the  C-R  bound. 
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related  with  those  of  the  covariance  parameters.  Hence,  the  C-R- 
efficient  regression  parameter  estimate  is  unaffected  by  the  lack  of 
knowledge  of  the  covariance  parameters.  (This  conclusion  remains 
valid  even  when  the  covariance  parameters  possess  no  unbiased  esti¬ 
mates.)  This  is  certainly  an  unsatisfactory  bound  in  view  of  the  fact 
that  the  optimum  estimator  of  a  was  found  earlier  to  depend  explicitly 
upon,  the  normalized  covariance  function,  and  therefore  on  3  for  this 
exanple.  This  motivates  an  application  of  Barankin's  theory. 

Ihe  Barankin  bound  for  unbiased  estimates  of  a  is  the  greatest 
lower  bound  on  the  mean  squared  error  of  such  estimates  about  sane 
pre-chosen  parameter  point,  ^  el\  ihe  choice  of  is  ccnpletely1 
arbitrary  as  long  as  it  lies  within  the  allowed  parameter  space. 
Whenever  at  least  one  unbiased  estimator  of  a  exists,  there  is  an  un¬ 
biased  estimator  of  a  which  achieves  the  Barankin  bound.  This  esti¬ 
mator  is  termed  "locally  best"  (for  £  =  . 

In  general,  only  locally  best  unbiased  estimators  exist.  Rao  [1] 
gives  a  necessary  and  sufficient  condition  for  the  existence  of  a 
uniformly  best  unbiased  estimator  (i.e.,.  an  MVUE).  Ihe  theory  of 
complete  sufficient  statistics  [39,  40]  offers  a  sufficient  condition 
for  the  existence  of  a  MVIE . 

Ihe  beauty  of  Barankin’s  theory  lies  in  the  fact  that  whenever  a 
uniformly  best  unbiased  estimator  exists,  the  Barankin  theory  will 
provide  us  with  it,,  even  if  no  complete,  sufficient  statistic  exists. 


In  our  application,  we  would  like  to  Uni,  1?  possible,  an 
estimator  of  a  *£iich  has  rlnlrtc  variance  aaaag  all  unbiased  esti— 
sates  of  a.  ¥e  oust  then  re®Ed  y  as  trie  true  value  cf  the  oarac- 

—  — O 

eter  j.  If  a  uniformly  best  irfciased  estimator  exists.  It  Mill  not 
depend  qpon  the  choice  cf  y^  and  we  Kill  hare  arrived  at  the  3KSE 

of  a. 

Actually,  we  will  show  by  direct  calculation  of  the  Bara&in 
estimator  -that  no  SWOS  for  a  exists  men  the  narsa Used  oyiariarroe 


Ancticn  has  unknown  paraneters.  Ms  can  be  shewn  mere  expeditiously 
by-  Baking  use  cf  the  fact  .that  the  (locally)  test  unbiased  estimator 
is  unique.  Applying  -this  Jfecfc  and- the  results  of -Chapter  U 

>*■  _i  ~ 

showed  that  a__  *  a  kSksi  ij  *  *  ,  and  that  is  an  usMased 
—do  — o  *  *  —CLS 

estimate  of  a  for  arbitrary  q,  we  see  that  o„_  is  best  for  a  if  and 

— ' "  "Utb  “*“ 

•1  ~  —1 

only  if  ii  *  *  -  That  is,  with  t*  =  $  is  the  Earas&in  esti¬ 
mator  which  is  best  for  y  *  y^.  But  this  requires  the  qctiaca  esti¬ 
mator  to  depend  qpen  S^;  so  the  BaranSdn  estimator  for  c  is  only 
locally  best. 


Despite  the  availability  of  the  sirple  argument  above,  it  is 


instructive  to  apply  BaranJdn's  theory  in  detail  to  the  joint  estima¬ 
tion  problem.  In  so  doing  we  will  develop  some  of  the  machinery 
which  is  useful  in  applying  Barankin’s  theory  to  the  estimation  of 


the  cgrarlarjce  pa raaeters. 


3-2  CfcaHer-Hao  Bouafe  Ter  Pirst-teder  Aatcregresslve  Gaeassiaa  Jfaise 

5be  problem  irrtredjced  as  Fxacple  1  earlier  is  actually  a  flrst- 

a 

order  autoregressive  scheme  .  "Ehis  permits  us  to  write  a  staple 
analytic  item  fer  the  inverse  cf  the  raoise  cosarlaraoe  matrix  if  the 
Hi  data  samples  are  uniformly  spaced  on  the  interval  [0,T]+.  lie  shall 
suppose  teat  this  Is  the  case.  ‘Shesi 


*  o  exp£-S  5jp-vj]  ;  y,v  « 


there 


x  »  T/{N-1) 

is  the  interval  between  time  samples. 


Mac  with  ij  -  *  “  and 


P  *  expCS  6) 
have  [£L] 


KM? 

~2 - T~  ~P.  (1  +  P  )  ~P 

Al  -  p)  ^  \ 
o  -*  1 


See  Section  4.3 

^Brennan  and  Reed  [55]  show  that  this  noise  covariance  aatrix  has 
a  staple  Inverse  even  for  unequally  spaced  tine  saaples. 
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Shat  is,  n  has  criSy  one  nonzero  eff-di^cnal  cn  each  side  of  the  sain 

7 

diagonal,  and  the  aein  diagonal  airies  are  1  +  p  except  f<r  the  two 
corner  eleoents,  which  are  unity. 

She  joint  profeahOity  density  function  cf  the  ceserrables  Is 

*  ~ *  “pj-  |H-  Ga]xn[R  -  Ck]  j  (3-6) 

(She  dependence  of  n  cn  3  has  not  been  explicitly  denoted.) 

Son  naaoe  the  following  associations  between  the-  cnpcpeaSs  of  y 
and  the  incnom  paraaettrs  £  and  a. 

X  *  (Ti  T2  T3]  -  IS  o2  a)  (3-7) 

and  define 


S2  lnp(Rjj)  { 
ay  ay  l 

■  n  * 


;  sa^i  *  1,2,3. 


(3-8) 

(3-9) 


Assuaing  that  all  the  ccrponents  of  y  have  unbiased  estimates, 

the  natrlx  L  2  cocprlses  the  desired  C-H  bond.  We  shall  denote  the 

eleoents  of  L  1  by  l*.  Ihen,  for  exaqple,  l11  is  the  C-R  bound  on 

the  variance  of  unbiased  estimates  of  g.  Ihs  C-R  bound  for  unbiased 

33 

estimates  of  a  is  given  by  1  .  'The  covariance  of  C-R-efficient  esfci- 

13 

mates  of  a  and  0  is  given  by  1  .  Similar  recalls  hold  for  the  other 
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elegants  of  lT\ 

ife  rats  proceed  to  calculate  the  elements  of  L  and  IT1, 
letting 

A(B;x)  *  23  P(H«jr) 

ve  haae 

A(Hjx>  =  3nj(2ar)"*/^J  +  |ln[dst(n3 

i  M 

Me  E£^  arrive  at  a  sisple  expression  for  the  determinant 
tMs  exanple: 

<**(„>  -  a«(l  -  p2)1-*  -  T-*(l  -  p2)2-* 

Also,  It  Is  casseaiert  to  define  the  quantities 
II 

«@>X>  »  £  VW  lr»  *  T3^]  Ir»  -  r3%] 
p,v=l 

«V  -  S  -  T*? 

|f=2 

K-l 

B(y3)  =  X  [r»  -  T3S,]  fVl  •  ^l1 
u=l 


(3-10) 

(3-11) 
f  n  In 

(3-12) 

(3-13) 

(3-1*0 

(3-15) 


'Then  for  1_,  we  have 
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32A(R;y)  -2  -2 

1=  _5 - - -  *  -  %  E  %  Ln[6et(n)]  -  ^  Q(R,X> 

^1  3S  33 


+  ^  p  7  V  [(i  +  6p2  +  p4)B(a)] 
o~(l-p2)3 


(3-16) 


Vat  observe  that 


A(a)  *  (tf-2)  o2 
KoT  *  fcJ-1)  o2p 


(3-17) 

(3-18) 


<rraei>2  *  (ik^s)2  '  °2 


(3-19) 


so  that 


i  _  52p2(k-D(3*p2)  ~  2o2p2(?:-1)  _  52o2(?;-l)(l^p2) 

'11  (1-p2)2  '■*  -2'2 


(1-P2)2 


(3-20) 


dhe  calculation  of  leads  to 


i  _  1 E  32ln[dst(n?1  3  9-’x) 

-12--21--26-  3g  3(o2)  ~  S3  5(o2) 


(3-21) 


The  first  terra  above  is  identically  zero  since  the  derivative  of 
the  determinant  of  n  with  respect  to  B  leads  to  a  function  which  is 


independent  of  a.  ibe  second  terra  is 
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Hence, 

i  «_^+IL=JL 

22  2a4  a4  2a4 

(3-30) 

Vfe  flm  1^  fran 

{  32A(R;x)  | 

1?-3"13^'E|3(o2)  J  23(o2)3a 

8^S®,x)  _  3  j  -1  J_  j2  +  {  „  )2 

*,*  3a  3“ioW)  1  1  *  ^ 

(3-31) 

+  (l+p2)A(a)  -  2pB(a)] 


*  -rNr  [eA  +  v* +  (1  +  p2)  £'  V» 

a  (1-P  )  L 


N-l  i 

*P  E  <eA +  wW 

y=l  J 

(3-32) 

is. 

*23  ‘  *32  S  ° 

‘  (3-33) 

fEhe  remaining  element  of  L  is  133. 

(82A(R;x)  )  -i  ( 32Q(R,X) ) 

**  I'H  3a*  f 

(3-34) 

32Q(R,y)  ,(  ,  r 

I?-'  L  1  '  +  (r«  '  “S»)SH 


Hence, 


,  S-l 

+  (i+p2)  jr 

N-l 

(r„  -  ^  -e  £  <r„  - 

y=2 

P=1 

N-l 

*|) 

-•>  E ( vi 

'  } 

o2(l-p2) 

77  7  W-l  o 

;i+4+(i  +  p>S^ 

V=2 

N-l  ,  N-l  _  -j 

-P 

'  E  4  -*  E  4*] 

Url  U=1 

*  22(1f  [g 

a2(l-p2) 

77  N-l  - 

?+4  +  (^-p)E41 

U-2 

* 


a2(l+p) 


97  N-i  » 

lgi +  % +  (1  - p)  E6*1 

U=2 


(3-35) 


(3-36) 


To  summarize  the  preceding  results,  we  shall  write  the  complete 


L  matrix. 


Q(R,X)  is  a  quadratic  form  in  R. 

—1  A  *2 
The  actual  bounds  are  found  from  L  .  If  we  let  B  and  o 

>  2 
denote  unbiased  estimates  of  their  respective  true  values,  B  and  o  , 

'  •  •  o  o 

then  the'  C-R  bounds  for  the  covariance  parameters  are 


v- 


N(l-p2)2 


(N-i)62pf[N(l-p2)^2p2] 
o  o  o 


(3-38) 


ri  J-\2 
(a  -oo)  > 


2aVl+p2) 

_ o ,  o 

N(l-p2)  +  2p2 
o  o 


(3-39) 


where  pq  is  given  by  (3-4)  with  B  =  B0- 

.  '  2 

The  covariance  between  C-R-efficient  estimates  of  B  and  a  is 

given  by 

2o? (1  -  p2) 

(3-^0) 


l12  = 


6[N(p2  -  1)  -  2p2] 


,12 


It  is  interesting  to  observe  that  1  ,  in  general,  is  not  equal  to 

zero.  This  indicates  that  the  C-R  bound  for  estimating  B  is  increased 
-  2 

when  the  noise  power,  a  ,  is  unknown. 

The  appropriate  results  for  dense  sampling  on  an  interval  of 


length  T  may  be  obtained  from  the  above  expressions  by  taking  the 
limit  as  N  00  and  6  -*■  0  in  such  a  way  that  (N-l)6  =  T.  This  yields 


3-3  Harankin  Bam 


2a  this  section  shall  ealcalate  tee  ^rsrMn  fcccrr3  Iter  estima¬ 
tion  cf  tee  c-eSaessIccaB  parasetar  a  when  tee  consrianoe  SaactSas  has 
aaSraaca  gssaneSags  S-  Sle  shall  ec£  need  to  spedi^-  ecs5Sg£t|y  what 
tee  gssaaeSaJc  dependence  an  _8  is  ira  erdsr  to  ceta£n  car  results. 

SdS  ^°,rrra  <wrrf -^g~y-~P ■=•  g«3e23ilSSy  tO  the  SnaHjSiS- 

2a  the  process  of  eaZroIating-  tie  Ssrsskiia  cccni  Iter  tasfcSased 
estimates  efews  reOl  develop  sane  erf  the  machinery  w&icia  ooala 
press  cseital  in  ecnstrnctlms  eeth-nalfes  cf  fens  mteanwn  csrisriasee 
paraneters- 

Sse  reader  is  referred  to  appendix  H  for  a  presentation  of  the 
theory  pertinent  to  tssls  section  aai  far  elaboration  on  the  essential 
osnsepts.  TJsfe  will  reseat  tee  taste  fanmalas  here  for  ccnneniance. 

let  y  denote  the  Jaimt  rector*  cf  tx&noan  paraasSers  belonging  to 
toe  parareoer  spare  a  as  in  (3-1)  -  fbs  Itnnchiam 

/p(Bjy)  pfcSjY*) 

nm^) -  ^  >  M’>^r  <3-«> 

is  central  to  the  tecrrnis'je  cctlin-d  in  ^gsenalx  H  for  caleelatisg 
toe  Earankin  frssssS.  Sere  is  an  arbitrary  fixe!  print  in  F,  and  is 
tie  paint  about  xnhen  the  Harsnkin  estimator  has  ninfcac  raean  snaarea 
error,  fe  shall  let  y  ccrresoond  to  the  tmse  value  cf  the  unknown 


°$(aoi)  =  J  {a±~  %!>  5  1  = 


Has  estbsEtar-  Talcs*  attafins  tfeSs  bam3  «in  be  dsictei  by  c^,  sal 

a.  Is  gives  by 
£ 


f  .o  ,  , 

=5,  *  J  5H7T  =¥£>  *  °oi  5  1  -  1»- 


Sfe  v*T»  neap  inraceed  to 


G{y,t'  It  )  »fcei!  p{Bjy)  is  as 


ejraessea  ira  (3-6).  fere  as  Kill  denote  esoIScitjsy  the 
*S  tram  She  parameter  vector  5  via  Sae  motatiaa  *»_. 

-  P 

fe  base 

[efetfo.)  «3st(n-,)j1/2 

GiY,y'  I Y  )  = - — - - -  T 

*x  lv  tefK&u*.  nin  ~ 

o 

^esp  -  ^  { [H  -  G*]Tn3iR  -  %I 

% 

*  [H  -  Ga’]Tn3,[H  -  Ge’] 

-  [R  -  [R  -  Go^}  cR 

o 

Jievf  define  the  following  matrices: 
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H  [r?g  t-  rjg.  ] 

o 


3=  ISe*]Te|s,  -  [SfcJTr] 


'3 


c  =  pe)Tfjp[%i  *  l&'lSp.ESs*]  -  I&s  ]xna  [&  ] 

a  —  c*  —  os  o 

—  o  — 


3323, 


(deSq  exzt-C/2) 

G^X.X*  !]()  * - S75 - - - i -75 -  2 

(2ar)S/2  (astr*  )'/2 
3o 


f ex?  -  A  fife  -  RT3T  -aj® 


(3-^3) 

(3-53) 

(3-51# 


(3-52) 


f  =  ^H1/2 

In  the  ahsns  integral-  'She  Integral  than  fceocces 

11  1 
—  C  1  *7*  •~T  -  ?  T  _  7  ^  _  9 

i  =  I  ex?  -  2®  K  -  p  3-53  *  R] .  [oetH]  d3 


(3-53) 


l 

[detH]  2  J'exo  -  j|j£t-ST1,2Jj^-^sr1,2^Tl  -  SH“ "rF  |  dR 

R  ^ 

^3 


=  [det  H]  2  exp[  |  (Si_3BT)] -(2«)N/2 


C3-5*0 
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xberefere. 


G{y,x3  lx, >  * 


(dgfeL)  ^{dsfai- .  )1/2ejp [-  ^  (C  -  S-33T)] 


(det  H)1'  2  (diet  !>„  )i/2 
S 

o 


(3-®) 


5Ms  eapresslca  is  valid  for  an  values  of  y  eF  whenever  3*  i-cas  in 

a  regtcn.,  F  (o-cencLIrg  crdy  :xkxi  g  ).  screen  ensures  the  existence 

.  o 

—1  ° 
cf  H- 

Jew  consider  she  ggrerr-l  fred  itoction 

dl.  (y)  *  c(S  -  3  )  c*(a  -a  )  o(cs  -  a  )  dy  ;  1  =  I,...,ra  (3-56) 
i  —  —  — o  i  si  x  *o  — 

1  i 

^feese;  a  ©sautes  the  vector  a  with  the  i—  earncrest  deleted,  5(.) 
i  ~ 

denotes  the  Dirac  disfcsiireitiaa,  sad  o’(-)  denotes  tie  first  derivative 
of  5(->  £371- 

Ifc  can  fee  verified  that  this  is  a  legitimate  choice  of  dA  in 
tesrs  of  the  existence  ard  equality  of  tie  integrals  preceding  Euua- 
ticn  (3-^5).  It  can  also  be  verified  that 


2  (GTne  G)±j  (Cj  -  coj) 

r«  5=1 


(3-57) 


2nd  (3-57)  holds  for  i  =  In  vectcr-nafcrix  form  we  have 

f  G^Y’Jy  )  dA(Y’)  =  [g\  G] [a  -  a  ] 


(3-58) 
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Iherefare,  the  desired  choice  fcr  aS  satisfying  (3-^5)  Is 

6aCy)  =  [G1^  G]-1  oACy)  (3-59) 


"She  Barsrkin  bound  for  estimating  the  1 —  eaqooraait  of  a  is: 


tfeing  (3-55)  we  find  that 


«B(® oi>  *  t®%  Gjil  (3-51) 

c 

Siis  agrees  exactly  with  the  result  in  Equation  (2-29).  Hence,  as 
mentioned  earlier,  the  untoatEss  of  the  locally  test  isaissed  esti¬ 
mator  implies  that  the  Earsnkin  estimator  for  c,  which  is  best  about 

the  true  value  of  y,  is  eoual  to  the  GIS  estimator  with  as  the 

3o 

weight  matrix.  Hiis  can  also  be  verified  by  direct  calculation  using 
Equation  (3-^7) -  fence,  we  have 


G]"1  /exp  -  1 1  [R-%]Tng[R-Ga] 

8o 

-  [RrGao]Tng  [RrGao]|  dA(X) 

—  o  — —  ) 


*  %  *  [GTng  G3-1GTng  [R  -  Ga^] 
o  o 

=  Ig\  G]-3gV  R 


(3-62) 
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3-S  Inplicatlcns  of  tbs  results  Obtained  so  far 

We  nave  shown  that  there  is  no  HvES  for  o_  when  the  nceralized 
noise  covariance  function  has  unkncwn  parameters,  8.  However,  we 
have  also  seen  that  in  sene  cases  there  is  a  dramatic  difference 
between  the  performance  of  the  KJBS  of  a  ana  the  IS  estimate  of  c.  It 
is  natural,  then,  to  ask  if,  in  the  case  of  unknown  covariance  param¬ 
eters,  there  is  sons  estimate  of  a  which  is  significantly  better  than 
the  IS  estimate,  though  naturally  not  as  good  as  the  M533.  I3h- 
fcrtunately,  it  is  difficult  to  laid  structure  to  the  problem  of 
searching  for  such  an  estimate. 

Che  approach,  however,  is  to  construct  the  Barankin  estimator  for 
a  which  is  best  about  an  estimate  of  the  trie  valise  of  3.  ibis  really 
accounts  to  using  the  CIS  estimator  for  a  with  an  estimate  of  £  used  to 
construct  the  appropriate  weight  matrix.  Biis  will,  not  necessarily 
lead  to  an  unbiased  estimate  of  a;  unless  the  estimate  of  £  is  made 
from  data  which  is  statistically  independent  frem  the  data  used  for 
estimating  a.  fionetheless ,  it  is  possible  that  the  bias  will  be  snail 
encuge  to  be  acceptable;  especially  if  the  mean  squared  error  in  esti¬ 
mating  a  is  significantly  reduced.  Moreover,  in  sane  cases  this  pro¬ 
cedure  will  provide  an  unbiased  estimate  of  a,  even  if  the  estimate 
of  8.  is  correlated  with  the  estimate  of  a.  ('ibis  will  be  elaborated 
upon  in  the  next  chapter.) 


This  leaves  us  with  the  problem  of  estieating  the  parameter  0. 
ife  could  use  the  raxinsn  likelihood  (ML)  estimate  of  if  it  is  ccn- 
veaient  to  go  so.  (Such  a  procedure  would  actially  amount  to  joint-HL 
estimation  of  a  ard  since  the  noise  is  Gaussian.)  Or  we  could 
esplqy  Bararsdirs  theory  to  estinate  g.  For  example,  we  could  perform 
a  dissection  of  the  parameter  space  for  0^  and  apply  the  technique  in 
.Appendix  H  to  obtain  an  inbiased  estimate  of  each  point  in  the  dis¬ 
section.  For  this  we  reed  only  specify  the  parametric  dependence  of 
p  en  g  and  use  (3-55)  fee?  G(y,y!{y  ). 

in  principle  either  of  these  techniques,  or  others  not  yet  men¬ 
tioned,  could  be  used,  thous*i  one  method  may  be  more  practical  than 
the  others  in  a  speci-ic  ease.  In  the  next  chapter  we  shall  consider 
an  estimate  of  (where  3^  denotes  the  unknown  parameters  in  the 
normalized  noise  covariance  function)  based  "upon  the  known  functional 

form  of  the  normalized  noise  covariance  function,  8(t,0)  =  fyi?^  . 

Ce(t)j2 

This  estimate  proves  to  be  remarkably  effective  for  iitproving  the  IS 
estimate  of  a  in  Examples  1  and  2. 


Chapter  % 

THE  ITERATIVE  ESTIMATION  PROCEDURE 

^.1  Introduction 

In  this  chapter  we  will  explore  a  particular  method  of  esti¬ 
mating  the  pertinent  covariance  parameters  in  Examples  1  and  2.  We 
will  then  incorporate  the  covariance  parameter  estimates  into  an 
estimate  of  the  regression  parameter.  Though  developed  thoroughly  only 
for  Examples  1  and  2,  the  metbGd  is  immediately  extendable  to  other 
problems  in  which  the  normalized  noise  covariance  function  contains 
unknown  nonrandom  parameters. 

Our  method  of  estimating  the  unknown  covariance  parameters  is 
based  ’pon  knowing  the  functional  form,  8(t,B)  =  4>(t,B)/4>(0,B),  of 
the  normalized  noise  covariance  function. 

We  will  use  the  IS  estimate  of  a  to  get  an  initial  estimate  of 
the  regression  function.  After  subtracting  the  estimated  regression 
function  from  the  data,  we  have  an  estimate  of  the  noise  process. 

This  estimate  allows  us  to  estimate  £  using  the  function  8(t,0).  We 
then  generate  the  estimated  normalized  covariance  function 
0(t)  =  0(t,|),  and  the  corresponding  matrix  0.  Finally,  we  construct 
the  GLS  estimate  of  a  by  using  n  =  0-1  for  the  weight  matrix.  At  this 
point  we  repeat  the  process  again,  this  tine  using  the  previously  con- 
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strutted  GS  estfraoe  cT  c  to  est£ggte  toe  rrise  process  and  tbsa  3- 
?Ms  revised  estisabe  c?  S.  leads  to  a  revised  estimate  cT  cr.  S5s  is 
t fee  iterative  esbfcnablaa  pra&dois- 

Sie  ■acJBl»'g  irrrsstS^te  tbe  and  Se  mesa  srrarsd  error*  «xT  t2e 
iberatlse  estbnate  cf  c.  He  bias  eaa  be  investigated  aralsbics22y 
to  g-r**3*  eztenh-  *vfrp>  mpgr**  sqpsrsd  error  will  be  lEnestEgstafl  by  dtlng; 
^.-r-vx— y  resrSts  c?  sSsnrZatlBg;  Scanpilss  3.  sad  2  era  s  digital  o ass- 


peter. 

fis  a  prelate  to  tfce  develrprenb  ®T  tbe  desired  estbnat&m  pro¬ 
cedure  Soar  gri  Star  She  cT  eanplstesess,  we  lsg2ode  a  briei* 
cf  tfee  ^alnt  argjgf’Tnam  21feaJlb®o3  C&E.}  estinstoir  Star  t-  2b 
wSll  to§  ?s¥v=n^  Shgb  ffriis  estimator  Is  mob  prarbiea!  Star  Sranplss  2  and 
2.  ife  j=»tcffa  presanb  so  estimate  c?  y  wMdia  Is  real;  2?  *3025*  aoSicab>2e  to 
aoxESgressive  sdbsnes,  bait  wSicsa  serves  to  motivate  tfee  tnrccedrre  we 
srplqy  to  estimate  S-  Isa  Etarnles  1  and  2- 
*3.2  ifexfrxn  IlkeSibSGd  Sstlnafclsm 


3e  HEd  esbSES be  cf  y  Is  dsflnei  as  tfee  solution  to  tbe  litelibcod 


eoaaticaa 


3A(B;1) 
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mets  ACRjy)  Is  as  lii  (3-10)  sad  £  denotes  the  HL  estimate  c?  Y- 


(*-l) 


T tr  1,  tS±s  leads  ts  tSs  system  cT  eq^sS&as 

siSErr)  a 

— - -  =  T*  rj  (v  —  esr  Jr-  =  j 

ca  **  pt?  S3  ~e>  o 

T=T  W^- 

j  _  s52cci-;a 

e3  I  TFT  Cl-P2) 


—  53  rS"  (r  —  Sr.  Mr  —  Sr  )  =  0 


y,v=l 


p»  p  ^  V  'U 


A  -  H  A  A  1 

ufee  p  Is  gi'sea  fry  G-^)  wlSs  8  =  3,  sa3  rj’^  =  —  I  S  =  g“ 

t£e  resdlt  ira  SimsSiaia  C3-5)  *e  m^y  rsacSSe  Snatiiara  1&-2)  as 


•• 

E(r  -  S?  jlfcj,  -  %_*>  *  *i %(l-32> 

C  _  £F« _ 


a-  TSsSi 


Sfe  -  ps  j)2  *  %u-?> 


*SMs  system  cf 


so  ofctaSm  tee  HL 


Eir=t  fce  solved  slmltaeegrjsls'  Ter  es  and  p 
2  cr  Y-  is  imaged  a  difilcsdfr  task. 


essi  cm  a  ©craste:*;  and  tee  situaticn  is  wassa  for  Exsrple  2. 


%-3  -£m  liggrcach  fer  £mtcregressive  Sdbsass 


Ssnocse  ve  iiave  data  cf  tfee  fem 


rB  =  *  % 5  v  =  1>— >!i 
5chb,  for  all  y 


P  W  W 


fere  v  derates  a  sarrsle  ef  a  zero  esss.  ifidLSe,  3s: 

p  -  33 


.xise  ac¬ 


cess,  and  t is  ep  ere  sarrZ-s  ef  a  Gazssisra  first  order  ssssregressire 
scneae.  let  e  denote  e  at  tee  igffela!  t±rs,  t  -  As  t  -*•  -*•,  a 

o  p  3  o  o  3 

sta&Ia  salisHcn  Sar  £&e  dHffaease  Ezsstlzn  (^-6)  exists  if  |pf<i. 
TMs  saEatisra  is  $2” 


e  -  E 

p  fc=0  t>_<: 

Observe  Shat  the  -variable  of  the  {e^J  process 

~  ■  OD  CD  CD 

2  2  v*  V  k  i -  x~*  2  2k 

°  =eP  =  TSSpp  WrSv  = 

fe=3  1=3  k=$) 

and  the  covariance  is 

#(u,v)  =  ee^  =  (pex  -f  w  )ev  =  pi(y  -  l,v) 
The  solution  to  (£-12)  is 


(1-p2) 


*  ,  !jj-v|  2  2  jy-vl 

o(y,v)  =  6yu  =  P1  1  ep  =  0  P!*  1 


fee?  Example  1  we  were  considering  toe  covariance  function 
$(y,v)  =  o  ero[-3o|y-v|] 
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12 


J?2  (ru  -  %)(Vi  - 


II 


v?2  (r»-l  0SB-1)‘ 


(*l-18) 


This  system  of  equations  Is  still  difficult  to  solve  simultan¬ 
eously.  However,  it  is  interesting  to  inspect  the  structure  of  c  and 
p  in  (4-17)  and  (4-18) . 

Sotice  that  c  in  (4-17)  Is  very  similar  to  the  ML  estivate  of  c 
in  (4-4),  and  if  p  were  known,  then  the  ML  estivate  of  a  would  be  the 
K55E  of  a. 

Ifctice  also  that  tie  estivate  of  p  in  (4-18)  amounts  to  sub¬ 
tracting  cut  an  estivate  of  the  regression  function  frar.  the  data,  ana 
using  the  known  covariance  function  of  the  noise  (see  (4-13)  and  (4-14)) 
to  construct  the  estimate.  In  this  particular  case,  a  Is  to  be 
obtained  frcra  the  simultaneous  solution  of  the  above  equations  and 
then  p  can  be  calculated.  However,  the  form  of  the  estimate  in  (4-18) 
suggests  using  the  IS  estimate  of  a  to  form  p.  ibis  then,  is  a  way  of 
motivating  the  scheme  of  estimating  the  covariance  parameters  by  what 
we  will  term  the  "inverse-covariance-function"  technique. 


4.4  Ihe  Inverse-Covariance-Function  Itechnique 

This  method  of  estimating  the  covariance  parameters,  is  a 
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sfopie  extension  of  the  estimate  In  Equation  (4-18) .  The  idea  Is  to 
obtain  an  estimate  of  the  noise  process  by  using  tbs  IS  estimate  of 
c  to  subtract  cut  an  estimate  of  the  regression  function.  ihen  we 
treat  the  estimated  noise  process  as  thougji  its  covariance  function 
had  the  itactlcnal  fora  given  by  $ (t,S)  -  In  the  sampled  data  case, 
for  exszple,  we  can  construct  estimates 

$Cpo)  =  Y,  V  ;  P  =  0,1,..., N  -  1  (4-19) 

y=pfrl  p  p  F 


where 


%ZZVV-aLS%  =  — 


(4-20) 


(Observe  that  ey  is  a  zero  mean  Gaussian  randan  variable,) 
We  than  set 


£(po)  =  6(p5,B)  J  (4-21) 

A 

and  solve  for  which  satisfies  (^-21) .  This  requires,  of  course, 
that  we  have  as  many  estimates,  $,  as  we  have  unknown  parameters  in 

Actually,  as  has  been  indicated,  only  parameters  in  the  normalized 
covariance  function  are  pertinent  to  inproving  the  IS  estimate  of  a. 
Therefore,  we  will  consider  estimates  of  the  normalized  covariance 
function,  8(t,B)  =  <J>(t,B)/4>(0,B.)  : 


4.4.2  Estimating  In  Example  2 

Suppose  in  Exacts  2  we  regard  the  bandwidth  parameter,.  as 
known,  and  the  cente^ffequency  parameter,  ac,  as  the  only  u^cnown 
normalized  covariance  parsreter.  .fere 

9(SjjO  =  e“®5  CC3  ta^o  (4-26) 

Again  we  have  only  one  unknown  parameter,  and  the  estimate  is 


=  i  ccs~1|e+55-9(6)| 


(^27) 


where  0(6)  is  given  by  (4-22).  (Actually,  it  is  sufficient  to  estimate 
the  quantity  t^S,  since  this  enables  us  to  construct  the  desired 
weight  matrix,  Q~x}  in  the  GiS  estimator.) 

4.4.3  Estimating  Bandwidth  and  Center-Frequency  in  Example  2 
If  we  treat' and  3  as  unknown  parameters,  we  need  two 
equations  to  obtain  estimates  of  these  parameters .  We  observe  that 

6(6,3)  =  e-85  cos  (4-28) 

0(26,3;)  =  e-2^  cos  2u  6  (4-29) 


©113  leads  to  the  equation 

~^-  =  0(6)  [l-tan2(w  6)] 
9(5) 


(4-30) 


Hence, 


A  _1  A  /N  n  1/2 

wc6  =  tan  1j[l  -  8(26)/  0(6)  z] 


( h— 31) 
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add 

A 

P  =  e”35  =  §(5)/cgs(u  5)  (4-32) 

c 

It  should  be  observed  that  in  the  above  examples  there  Is  a  pos¬ 
sibility  that  (4-25),  (4-31)  and  (4-32)  will  lead  to  illegitimate  esti¬ 
mates,  for  in  (4-3 1)  we  have  no  assur&ee  that  8(26)<[0(5)]2,  and  In 
(4-25)  and  (4-32)  we  cannot  be  assured  that  p>b.  Furthermore,  since 
these  estimates  will  be  corputed  on  a  digital  cccputer.  It  is  wise  to 
modify  the  resulting  equations ,  whenever  necessary,  to  prevent  the 
occurrence  of  illegal  arithmetical  machine  operations.  For  example, 
in  (4-31)  we  can  be  sure  that  we  take  the  square  root  of  a  ncn- 
negative  nixrfcer  if  we  take  the  absolute  magnitude  of  the  quantity 
Cl  -  6(26)/{o(6^2] .  ’While  this  must  be  regarded  as  an  artifice.  It 
will  prevent  a  machine  abort  and  provide  an  answer,  which  can  then  be 
subjected  to  scrutiny.  (Of  course,  one  could  provide  for  an  indica¬ 
tion  by  the  computer  whenever  an  illegal  operation  would  have  re¬ 
sulted  if  not  for  the  artifice.)  It  is  remarked  here  that  in  the 
simulations  of  Exauples  1  and  2  such  artifices  were  not  required. 

4.5  Ihe  Bias  of  the  Iterative  Estimate  of  a 

Once  the  parameters  of  the  normalized  covariance  function  are 
estimated  (by  any  method)  we  generate  the  desired  weight  matrix. 


<  c* 
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=  e*1,  and  the  first  iteration  of  the  procedure  provides  us  with 


A  —j  Ta 

Or  *  [G‘nG]  Vim 
'1 

Notice  that  the  mean  of  a,  is 

^1 


(4-33) 


=at  [GTnG]  ^rye 


(4—34) 


We  shall  now  investigate  the  bias  vector. 


b  =  [GTrfel  Vhe 


(4-35) 


-1 


Consider  any  two  nxn  matrices,  A  and  B,  such  that  A  “  exists.  We 


can  express  the  matrix  [A  +  B]  1  as  a  power  series 


[A4B]'1  =  A  1il  -  BA~X  +  (BA"X)X  -  (BA-X)J  +  ...] 


-1  .  2 


.-1x3 


(4—36) 


where  I  denotes  the  nxn  identity  matrix,  (Bauation  (4-36)  is  easily 
n 

verified  by  separately  pre-  and  post-multiplying  both  sides  by  [A+B] . ) 
Now  rewrite  (4—35)  in  the  form 


=  (g\gJ 


+  Gx(n  -  5)G 


l"1 

3  G  rye 


( 4—37) 


T  —1 

where  £  is  any  NxN  matrix  for  which  [G  £G]  exists.  If  we  associate 
[GT£G]  with  A  in  (4-35)  and  [GT(n  -  ?)G]  with  B,  then 

.-A 


b  =  tG^Gf^jj^  -  ^[GT  (n  -  ?)G]  [GT^G] 


♦  l 


EGT(n  -  c)0] [GT5B]_1 


\1 


G  ne 


(4—38) 


New  suppose  that  for  every  pair  (p,v)  where  KjkN,  IsvcM,  n 
.can  be  expressed  in  the  form  -  - 


yv 


a  : 

n 


qi 


^•££r% 


i  fc=l 


where  q^,  is  spy  even  integer  £.N,  the  are  any  real  numbers*  and  £ 
-is  apy  (real)  n&nber  (it  may  be  a  random  variable)  which  is  a  common 
factor  of  all  the  elerents  of  n<  This  means  that  every  element  of  n, 
normalized  to  it,  has  an  expansion  into  a  weighted  sum  of  products  In¬ 


volving  an  even  raarfcer  of  factors  of  sero-me_n  Gaussian  random  var¬ 
iables.  For  instance,  consider  Exanpie  1.  Fron  Equation  (3-5)  it 
followl  trett  with  $  as  m  f.^251,  n  satisfies  (4-39). 

She-  significance  of  the  property,  in  (rl— 39)  is  that  when  it  holds, 
every  term  In  \ .  ♦-  .j  ip.  Equation  (4-38)  involves  a  linear  eaiMnation 
of  products,  in  which  there  are  an  odd  number  of  factors  of  zero-mean 
Gaussian  .random  variables.  Hence  when  (4-3?)  holds,  b  =  £,  ana  the 
first  iterated  estimate  is,  unbiased  for  xx;  it  is  easy,  to  see  that 
whan  ( -M— 39 )  holds,  each  successive  iteration  will  also  yield  an  un¬ 
biased  estimate  for  a. 

Hence,  the  iterative  estimate  for  a  in  Exanpie  1  is  unbiased. 

When  (i|_ 39)  does  hot  hold  it  is  difficult  to  establish  the  bias 
of  the  iterative  estimate  for  a.  However,  (4-39)  merely  comprises  a 
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sufficient  condition  for  unbiasedness.  It  is  therefore  possible  that 
even  If  (4-39)  is  not  satisfied  the  iterative  estimate  for  a  will  be 
unbiased.  Also,  the  bias  may  be  small,  even  if  it  is  not  exactly  zero. 
In  Example  2,  for  instance.  Equation  (4-39)  is  not  satisfied  when  w 
is  treated  as  an  unknown  parameter,  Nonetheless,  as  will  be  seen  from 
the  simulation  results,  the  bias  of  the  iterative  estimate  for  a  is 
insignificant. 

4.6  Simulation  Results 

We  will  hew  move  to  a  discussion  of  the  results  of  simulating 
Examples  1  and  2  and  employing  the  iterative  estimate,  of  a.  We  have 
already  described  the  estimation  procedure,  and  we  have  given  the 
explicit  form  of  the  covariance  parameter  estimates  for  each  example. 

The  simulation  naturally  involves  a  Hcnte-Carlo  approximation  of 
the  desired  ensemble  expectations.  A  technique  suggested  by  levin  [43] 
was  used  to  simulate  the  desired  noise  randan  process .  The  actual 
computer  programs  are  presented  in  Appendix  III. 

It  was  decided  that  since  the  mean  and  variance  of  the  IS  esti¬ 
mator  could  be  calculated  analytically,  a  comparison  of  the  analytical¬ 
ly  obtained  results  with  the  simulation  results  for  these  quantities, 
would  serve  as  an  indication  of  the  reliability  of  the  simulation. 

The  simulations  were  carried  out  .using  a  sanpled  data  approach 
with  ten  time  samples  uniformly  spaced  on  an  interval  of  unit  length. 


! 


i 
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The  results  of  1000  Hcnte-Carlo  runs  (requiring  just  over  two  minutes 
of  IS!  3o0  naming  time  for  each  case  above)  are  presented  in  Table  I. 

The  simulation  for  Example  2  is  slightly  more  complicated  because 
the  matrix  inverse  of  0  does  not  have  the  simple  structure  it  had  in 
£sa~ale  1.  A  computer  program  for  matrix  Inversion  was  used  to 
calculate  9  1j  and  two  iterations  were  performed  to  estimate  a.  The 
corputer  running  time  was  limited  to  ten  minutes  for  each  of  the  two 
cases  tried,  and  the  maximum  number  of  Konte-Carlo  trials  which  could 
be  obtained  within  this  time  period  was  used.  As  in  Exanple  1  the 
simulation  employed  ten  time  samples  uniformly  spaced  on  an  interval 
of  unit  length.  In  case  1  only  the  center-frequency  was  regarded  as 
the  unknown  covariance  parameter,  and  Equation  (4-27)  was  used  to  esti¬ 
mate  it.  In  case  2,  both  the  bandwidth  and  center-frequency  were 
estimated  using  Equations  (4-31)  and  (4-32).  In  both  simulations  we 
used  a  =  2.0,  B  =  .01,  and  f  =  w  /2tt  was  set  at  1.5  hz  cor- 
responding  to  k  =  1  in  (2-51).  The  results  of  the  simulation  are 
displayed  in  Table  II. 
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.  4.7  Discussion  of  Simulation  Results 

With  the  data  conveniently  sunmarized  in  Tables  I  and  H  we  can 
make  several  observations .  ELrst  we  will  discuss  the  data  in  Table  I 
pertaining  to  the  autoregressive  example. 

Notice  that  in  all  but  Case  4  there  is  a  dramatic  difference 
between  the  variance  of  the  LS  estimate  and  the  MVOS;  even  though 
these  simulations  are  for  the  sanpled  data  case.  This  indicates  that 
the  conditions  for  potential  dramatic  improvement  over  the  LS  estimate 
given  in  Equation  (2-37),  which  were  derived  on  the  basis  of  a  con¬ 
tinuous-time  approach,  also  serve  as  a  guide  to  such  behavior  for  the 
* 

sampled  data  case.  In  fact.  Case  4  does  not  satisfy  the  conditions 
in  (2-37),  and  indeed,  does  not  offer  an  example  of  a  case  where 
dramatic  improvement  over  the  LS  estimate  is  possible. 

As  mentioned  earlier  a  comparison  between  and  aQ,  and  between 
and  MSEls  serves  as  an  indication  of  the  accuracy  of  the  simula¬ 
tion.  With  this  in  mind,  there  is  little  doubt  that  the  iterative 
estimator  for  a  significantly  improves  the  LS  estimate,  in  problems 

# 

Undoubtedly  the  extent  to  which  this  holds  depends  upon  how  "dense" 
the  sampling  is.  For  the  cases  studied  herein  the  sampling  rate  was 
sufficiently  high  that  all  of  the  samples  fell  within  the  correlation 
»  time  of  the  noise  process  (the  value  of  x  required  for 

4>(t,0)/<KO,£)  =  e"1). 
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characterized  by  Exanple  1,  whenever  dramatic  improvement  is  possible. 
Even  in  Case  4,  where  dramatic  improvement  is  not  possible,  a  com¬ 
parison  of  and  MSEIT  indicates  that  the  iterative  estimator  is 

no  worse  than  the  LS  estimator. 

Also,  the  degree  of  improvement  is  such  that  the  variance  of  the 
iterative  estimate  of  a  is  often  within  a  few  percent  of  the  variance 
of  the  MVUE  of  a  after  only  two  iterations.  In  Cases  1  and  2  this 
represents  a  reduction  in  the  variance  of  the  13  estimate  of  two 
orders  of  magnitude.  In  Case  3  one  order  of  magnitude  reduction  in 
variance  is  obtained;  and  this  is  all  that  could  possibly  be  obtained 
in  view  of  the  ratio  VLg . 

Coupling  the  above  results  with  the  fact  demonstrated  earlier 
that  when  the  inverse-covariance  function  estimator  is  used  for  p  the 
iterative  estimator  of  a  is  unbiased,  we  conclude  that  the  boot¬ 
strapping  technique  for  estimating  a,  using  the  inverse-covariance 
function  estimator,  is  an  effective  tool  for  estimating  the  regression 
parameter  in  problems  similar  to  Exanple  1. 

An  inspection  of  Table  II,  which  pertains  to  the  simulation  of 
Exanple  2,  reveals  the  same  outstanding  performance..  Note  that  even 
when  the  covariance  function  has  two  parameters  to  be  estimated,  the 
iterative  estimate  of  a  achieves  two  orders  of  magnitude  reduction  in 


the  mean  squared  estimation  error;  and  again  after  only  two  itera¬ 
tions.  It  should  also  be  observed  that  the  mean  value  of  the  iterative 
estimate  of  a  is  approximately  equal  to  aQ.  Hence,  even  though  the 
condition  in  (4-39)  is  not  satisfied  for  Example  2,  the  iterative 
estimator  for  ct  has  negligible  bias. 

It  is  interesting  to  observe  that  in  both  of  the  examples  sim¬ 
ulated,  the  bootstrapping  estimation  procedure  is  capable  of  reducing 
the  mean  squared  error  of  the  LS  estimator  by  orders  of  magnitude 
even  after  only  one  iteration. 

4.8  Conclusions 

We  have  demonstrated  that  in  problems  involving  the  estimation 
of  linear  regression  parameters  in  colored  Gaussian  noise,  the  simple 
LS  estimator  can  be  significantly  suboptimal .  When  the  noise 
covariance  function  can  be  described  as  a  known  function  of  a  finite 
number  of  unknown,  nonrandom  parameters  it  is  possible  to  take  ad¬ 
vantage  of  this  information  to  improve  upon  the  15  estimator. 

By  starting  with  the  LS  estimator  of  the  regression  parameter  and 
employing  an  iterative  bootstrapping  procedure,  we  have  shown  that  it 
is  possible  to  greatly  reduce  the  mean  squared  estimation  error,  even 
after  only  one  iteration.  Furthermore ,  even  though  no  MVUE  for  the 
regression  parameter  exists  unless  the  normalized  noise  covariance 
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function  is  known  precisely,  we  have  seen  that  the  performance  of  the 
iterative  regression  estimate  is  very  near  to  the  performance  of  the 
MVUE  after  only  two  iterations. 

Depending  upon  the  noise  covariance,  the  bootstrapping  procedure 
may  lead  to  an  unbiased  estimate  of  the  regression  parameter;  or  to  an 
estimate  which  is  approximately  unbiased.  When  the  noise  is  a 
Gaussian,  stationary,  first-order  autoregressive  scheme,  the  iterative 
estimator  can  be  rendered  unbiased. 

The  bootstrapping  procedure  requires  an  estimate  of  the  pertinent 
covariance  parameters.  We  have  suggested  several  approaches  which  may 
be  taken  to  obtain  the  desired  estimates: 

1)  maximum  likelihood 

2)  Barankin 

3)  inverse-covariance  function. 

Judging  from  the  simulation  results,  it  does  not  appear  that  the 
success  of  the  bootstrapping  procedure  requires  an  unbiased  estimate 
of  the  covariance  parameters.  To  this  extent  any  of  the  above-men¬ 
tioned  approaches  to  covariance  parameter  estimation  are  admissible, 
and  the  Barankin  approach,  which  would  provide  an  unbiased  estimate 
at  least  for  selected  points  in  the  covariance  parameter  space,  is 
possibly  superior.  In  the  examples  simulated,  the  inverse-covariance 
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_  function  technique  was  employed  because  of  its  simplicity. 

4.9  Suggestions  for  Further  Study 

We  have  exhibited  a  class  of  problems  in  which  the  IS  estimator 
of  linear  regression  parameters  can  be  significantly  improved  upon 
using  a  bootstrapping  procedure.  This  class  of  problems  is  charac¬ 
terized  by  the  property  that  the  noise  spectrum  contains  a  "spike" 
which  has  a  bandwidth  which  is  small  ccnpared  to  the  regression 
signal's  bandwidth,  and  which  is  located  so  that  the  "mainlqbes"  of 
the  signal  and  noise  spectra  are  separated  in  the  frequency  danain. 

It  seems  very  difficult  to  state  useful  necessary  and  sufficient 
conditions  for  the  effectiveness  of  the  bootstrapping  procedure  (or 
any  other  procedure,  for  that  matter).  It  would  be  desirable,  however, 
to  extend  the  results  obtained  to  other  problems  where  the  noise 
spectrum  contains  several  spikes,  for  example. 

It  would  also  be  interesting  to  find  other  classes  of  problems 
which  have  the  potential  of  dramatic  improvement  over  the  IS  esti¬ 
mator,  i.e.,  where  the  variance  of  the  MVUE  is  significantly  smaller 
than  that  of  the  IS  estimator. 

Another  topic  of  interest  is  related  to  the  complicated  problem 
of  investigating  the  sensitivity  of  the  GLS  estimator  to  the  choice 
'  of  the  weight  matrix  [44].  It  would  be  interesting  to  compare  the 
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In  Section  2.4  we  had  need  for  the  solution,  h(t),  of  the  in¬ 
tegral  equation 
T 

o2f  h(t).  exp(-B|t-s|)  cos [w  (t— s) ]  dt  =  1  ;  se[0,T]  (1-1) 

0  c 

Here  we  present  the  details  for  finding  h(t). 

According  to  Zadeh  and  Raggazzini,  [45,  46],  by  virtue  of  Equa¬ 
tion  (2-62)  the  solution  is  of  the  form: 

h(t)  =  Koe"kt  +  K  +  K26(t)  +  K3S(t-T)  +  K4e~k(T_t)  (1-2) 

We  will  find  the  six  constants  by  direct  substitution  of  (1-2)  into 
(1-1) .  To  facilitate  the  algebra  we  will  perform  the  integration  for 
each  term  of  h(t)  separately. 

_lcfc 

Hence,  integrating  with  the  kernel  KQe  we  obtain, 

T 

I.  =  a2f  K  e”kt  e'S*t"sl  cos[co  (t-s)J  dt 
1  0  0  c 


=  a2K  f  e-kt  e_^s  cos[u  (s-t)]  dt 


"0 


+  a2 Kq  /  e"kt  e+e(s-t)  cos[u>c(s-t) ]  dt 


=  I11  +  I12 


(1-3) 


85 


(1-10) 


To  solve  for  K  and  K.  we  equate  the  coefficients  of  e  sinw  s 
o  4  c 

and  e  sin  [to  (T-s)]  to  zero.  This  results  in  the  equations 


K  w  K.m  K.o)  e  ^ 

o  c _ +  1  c  +  4  c _ 

w2  +  (k-3) 2  w2  +  B2  w2  +  (k+3)2 


Kue  K.u  K.to 

o  c _  .  1  c  ,  4  c _ 

u2  +  (k+3)2  w2  +  32  w2  +  (k-3)2 

c  c  c 


This  system  of  equations  yields 


K  =  K. 
o  4 


1  e“K1  1 

K  -x - - - x  +  - x  +  K,  ?  X  9  =  0 

w2  +  (k-3)2  a>  +  (k+3)2  1  w2  +  32 

c  c  c 

But  from  the  constant  terms  on  both  sides  of  (1-1*1)  we  have 
ui2  +  B2 

Ki  =  -Hr~ 

23o 

Inserting  this  result  into  (1-19)  yields 


[w2  +  (k-3)2][w2  +  (k+3)2] 

0  23a2  ^[w2  +  (k+3)2]  +  [w2  +  (k-3)2]e“kT^ 

Finally,  setting  the  coefficients  of  e”^scosiocs  and 
e~3 (T-s) cosm  (T-s)  equal  to  zero  provides  us  with  exactly  the  same 
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equations  for  K2  and  K3;  so  that 

K2  =  K3 

and 


K02(k-3) 
w2  +  (k-B) 2 


-  I  +  V5 


K  02(k+e)e~kT 
o _ 

u2  +  (k+3)2 


Hence, 

k([(u2  +  (k+3)2]  -  [w2  +  (k-3)2]e"kT) 

Kz  23a2([co2  +  (k+B)2]  +  [to2  +  (k-3)2]e“kT) 


(1-22) 


(1-23) 


(1-24) 


Using  (1-15)  these  results  may  be  expressed  in  the  form  of  Equations 
(2-55)  through  (2-59). 
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APPENDIX  II 


In  Chapter  III  we  made  use  of  Barankin’s  theory  of  locally  best 
unbiased  estimation.  Here  we  give  details  of  the  development  of  this 
subject;  following  closely  the  approaches  of  Barankin  [1*1]  and  Swerling 
[15,  *171. 

We  begin  by  defining  a  measure  space  (X,?^)  where  X  is  a  sanrole 
space  of  points,  x,yis  a  cr~field  of  subsets  of  X,  and  y  is  a  countably 
additive  measure  defined  on?1'. 

Consider  the  family  of  probability  measures  |Pa;aeA|.  defined  on 
y  where  a  is  a  real  scalar  parameter  belonging  to  the  set  A.  We  shall 
assume  that  the  measures  PQ  are  absolutely  continuous  with  respect  to 
the  measure  y  for  every  aeA.  Hence,  we  may  define  the  family  of 
probability  density  functions,  <P  =  {pQ;cxeA|,  on  X  with  respect  to  the 
measure  y  using  the  Radon-Nikodym  theorem.  So,  let  B  be  ary  subset  of 
X.  Then 

/  dP  =  /  p  dy  V  BCX  and  ask  (II-l) 

B  a  B  a 

In  our  application  we  shall  take  y  to  be  Lebesgue  measure,  and  (P 
will  be  the  family  of  Gaussian  probability  densities  defined  on  the 
extended  real  line.  We  shall  retain  the  above  notation,  however,  for 
the  sake  of  generality. 

Our  goal  is  to  draw  inferences  on  the  unknown  nonrandom  parameter. 


1  *-**M'~ 


I 
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a,  or  on  seme  known  (real  valued)  function  g  of  this  parameter.  That 
is,  let  g  be  a  knewn  real  valued  function  defined  on  A.  We  seek  a 
real  valued  p-meas unable  function,  Tq(x) ,  cnX. which  is  an  unbiased 
estimate  of  g(a) ,  and  which  is  best  in  the  mean  square  sense  at  sane 
prescribed  point  a .  =  aQ.  Thus,  if  we  let  *U denote  the  class  of  all 
p-measurable  functions,  T(x) ,  on  X  having  the  unbiasedness  property. 


/■ 


mPa  -  g(o) 


Ycsz A 


(II-2) 


we  seek  T  e  U  such  that 
o 

A  -  s(oo)]  2  ®a  <ftp  -  g(o0)i  2  dPa  We# 
X  0  X  ° 

It  is  convenient  to  define 

<Kx)  =  T(x)  -  g(a  ) 


h(a)  =  g(a)  -  g(aQ) 


(II-3) 


(II-4) 

(II-5) 

We  also  define  the  norm  on  X  with  respect  to  the  nominal  measure 


Pa  for  any  p-measurable  function,  ^  as 
II  '/'ll  =  j^/^OOdP^  (x)] 


a  j 
o.  -* 


(XI— 6) 


Note  that  ||  <J>  |f  is  the  mean  squared  error  of  the  estimate  T  about 


s(aQ). 


Now  for  TeU  we  can  rewrite  (II-2)  as 
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■>;)dP  (x)  =  h(d)  5fosA 

CL. 


(n-5) 


That  Is,  T(x)'  is  ah  moiaseii  es^ismtdr  Sqp  g(d)  if  ana  only  if  $(x)  is 
ah  unbiased  estimator  for  hCa) . 

He  shall  how  Sssuse  that  each  of  the  ratios 


I>Jx) 


■»  \ 

liia.xj  -  ~ — - 


%  w 
o 


(n-8) 


is  -defined  y-al|bo3t  everywhere  an  X,.  and  that  7r(a,x) 4  considered  .as  a 
function  of  a,  is  Eor-el  measurable.  Then  Equation  (H-T)  takes  the 
fora 


/^(x)7r(a,x)dF  (x)  =  h(a)  j  3feeA 
X:  ao 


(H-9) 


Let  us  now  consider*  the  measurable  space  (A fr),  and  let  X  be  any 
signed  measure  on  (A,4)  such  that  the  integrals  /h(a)dX(a)  and 


j  7r(a;x)dX(a)SiP  (x).  exist.  Then  we  may  integrate  both  sides  of 


a 


(n~$)  with  respect  to  X  and  obtain 


|  /h(«)dX(a) 

/jfw(a,x}(t>(x)dXCa)dP'  (x) 

j  A 

AX  % 

x)dX(a) 


(n-io) 


93 


(a) 

To  this  end,  for  any  given  aQ  let  2/2  denote  the  set  of  all 

functions,  6(:<)  satisfying  (A-7)  for  which  |j$>{]2  <  «.  Assure  that 

(a)  “ITT  (a  ) 

*l)i1  is  nonempty.  Let  71 2  denote  the  closure  of  71 2  taken 

2  ~1ST 

with  g  [J  as  the  metric.  Since  71 2  is  closed  and  convex,  there 
exists  a  function,  4>q(x),  (and  corresponding  to  it,  Tq(x))  which 

2  ~1«J 

Eimmizes  |j$fl  for  all  functions  in  71 2  and  which  is  unique  with 

probability  one  (P  measure  for  any  aeA) .  It  is  remarked  that  in 

general  Tq(:<}  will  depend  upon  aQ.  This  will  be  the  case  unless  Tq(x) 

2 

minimizes  gijj  for  every  a0e** 

It  will  be  assumed  that  Tq(x)  is  an  unbiased  estimate  of  g(a) , 
for  if  this  were  not  the  case,  there  would  be  no  unbiased  estimate  of 
g(a)  widen  had  nriniirirt  mean  squared  error  when  o  =  aQ. 

Consider  no-//  functions  f(a)  defined  oyer  A  having  the  form 

f(ci)  =  /H(x)dP  (x)  yoeA  (II-12) 

•J  o 

X 

where  H(x)  is  any  pleasurable  function  such  that  fiHlj2  <  ®.  Let  F 
denote  the  family  of  all  such  functions.  Then,  it  is  easy  to  show 
that  for  any  feF  and  for  a ny  concomitant  H,  the  integral 
JV(x}H(x)dPe  (x)  yields  the  ssse  constant.  To  see  this,  for  sqk 


9*1 


4;,  (real)  e  define  the  u-measurable  function 
T*(x)  =  To(x)  +  e[Hx(x)  -  H2(x)J 


(H-13) 


where  H1  and  H2  both  satisfy  (n-12)  with  ShJI2  <  «  and  ||H2|2  < 

Since  and  H2  satisfy  (IT-12) a  T  (x)  is  in  71.  The  fact  that  ||HX|[2 


(«  ) 


and  |h'2[jz  are  bounded  ensures  that  T  (x)  is  in  °  •  Consider 

D(e)  =  Jfj. T*(x)  -  g(a  )]  2  dP  (x) 

X  ° 

Calculating  -  will  reveal  that  unless 

’  d  £ 

/ To(x)[H1(x)  -  H2(x)]  dPa  (x)  =  0 


(11-1*0 


(II-15) 


we  will  have 


3D(g) 
3  e 


f  0.  This,  however,  would  contradict  the 


e=o 


definition  of  T  (x) . 

o 


We  nay  new  define  a  functional  A  on  F  as  follows.  For 


£(a)  =  J*H(x)dP  (x)  ;  osA,  ||Hjj2 
X  a 


<  CO 


define 


A  °  K3  =  /h(x)[to(x)  -  g(aQ?J  dP^  (x) 


(11-16) 


X  L 


(a  ) 


The  fact  that  J* T  (x)K(x)dP  (x)  is  a  constant  ensures  that  A  °  [f] 
X  ° 


is  uniquely  defined  for  any  fsF  and  for  any  concomitant  H.  Note  that 
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and  assume  that  G(a,a'  I  a  )  is  bounded  for  all  aeA  whenever  a'  belongs 

•  o 

to  some  set  ACA.  It  is  easy  to  see  that  G(a,a'  |a  ),  considered  as 
a  o 

o 

a  function  of  a  can  be  expressed  in  the  form 

G(a,a'  |a  )  =  /H(x)dP  (x)  ;  ||h||< 

°  X  a 

if  H(x)  is  identified  as 

H(x)  =  Tr(a' ,x)  (11-20) 

and  if  |jH||2  <  00  whenever  a'eA  . 

o 

(a) 

Hence,  for  A  operating  on  G  considered  as  a  function  of  a,  we 

have 


A  0  [G(a,a‘ |aQ)]=  ,x)  Jto(x)  -  g(aQ)J  dPQ  (x) 


=  g(a')  -  g(a  )  ;  a'eA 


(II-21) 


New  suppose  for  the  moment  that  A  is  composed  of  discrete  points 

{a±  ;  i  =  0,1,..., n}  (n  may  be  infinite).  Also  suppose  that  g(a)  on  A 

can  be  expressed  as  a  linear  combination  of  the  functions  G(a,a' |aQ) 

considered  as  functions  of  a'  for  a'eA  .  Ihat  is,  suppose  there 

o 


exists  sons  set  of  real  numbers  ja.  |  such  that  for  {o^j-eA 

n  1 

g(a±)  -  g(a0)  =  £  a^GCa^So,)  ;  {a^eA^ 

J—O  o 


(11-22) 


(The  set  |ai|  may  be  found  by  inverting  the  (n  +  1)  x  (n  +  1)  matrix 
of  elements [G(a±>aj|ao)  ;  i,j  =  0,1,..., n]  ).  Then 

9  )  (q  )  n  (ct  )  p  -» 

I*0I  =  A  [g]  =  A  [g  "  g(aQ)]  =  j?ajA  °  [G(“i>aj  la0)J 

=  S  aj  [g(aj)  -  g(a0)J  (n-23) 

Equation  (11-23)  provides  us  with  an  exact  expression  for  the  Barankin 
bound  when  the  parameter  space  is  ccnposed  of  a  denimerable  set  of 
points. 

We  can  verify  that  the  estimator 
n 

4>(x)  =52  a  ir(a  3x)  ;  Ja.leA  (11-2^1) 

j=o  3  J  1  ao 

is  unbiased  for  h(a)  when  the  |a.J  satisfy  (11-22),  since  for  c^eA 


n 

=  =  gCc^)  -  g(%)  ;  {ai}eA  (H-25) 


t  I 


r -, 


^^«Mirf*t(!P(Wi«f 


■>  { ■ 
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|^(x)|2  =  ft  t&] 


\iKcij  Jx)iT(ak.x)p(x;a0)dx 


n 


n 


n 


E  aiEakG(a  Ja,  |a  )  =  £a[g(a 
j=o  Jk=o  k  3  *  °  j=0  jL  • 

.>]■ 


) 


j=o  Jk=0  -  J  -  -  f=~  Jl  5 

-  s(“0)j 

Hence,  by  the  uniqueness  property  of  the  Barankin  estimator 


(11-26) 


n 


Te(x)  =  4>n(x)  +  g(ctj  =  2  a^Ca^x)  +  g(aQ) 


f-'  1 

j=o  J 


(11-27) 


So  far  we  have  shown  how  to  calculate  the  Barankin  bound  and  the 
Barankin  estimator  -when  A  is  composed  of  a  countable  set  of  points. 

Vfe  now  want  tq  generalize  these  results  to  the  case  when  A  is  an  in¬ 
terval  on  the  real  line .  Loosely  speaking,  what  is  needed  is  a 


representation  for  g(d)  on  A,  similar  to  the  expression  in  (11-22); 

-  ■  .  (a) 

so  that  the  linear  functional  A  °  operating  on  g(a)  can  be  expressed 

.  (cc  5- 

ih  terms  of  A  °  operating  on  G(a,a’ja  )  considered  as  a  function  of 


q;  Ki.rst  we- present  some  preliminaries. 

Suppose  that  f  (a)  is  any  function  defined  chi  A  which  belongs  to 

2 


?i  Suppose  that  for  every  i  there  is  sane  H^x)  with  jjHj  <  «>  such 
that  with 


ft(cc)  =  jf Hi(x)dP<t(x)  ;  cteA 


(11-28) 


7. 
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100 


That  is,  g(a)  -  g(aQ)  has  the  representation 


g(a)  -  g(ao)  = 


III  ^ 


,x)dX(a') 


aP  (x) 
a 


(H-3^) 


Identifying  f 7r(ct’ ,x)d£(a' )  with  H(x)  in  (11-12),  we  see  that  if 


j|  J j^dACa1)!2  <  °°j  then  [g(a)  -  g(aQ)]eP,  and  A 


(o  ) 
o  | 


is 


uniquely  defined.  Therefore, 
(a)  _  («  ) 


=  A  °  [g]  =  A  °  [g  -  g(aQ)] 

=  // ir(a*,x)dX(a«)J^To(x)  -  g(ao)JdPQ  (x) 

X  a 

o 

=  //[To(x)  -  g(ao)]dPa,(x)dA(a«) 


A  X 
a 

o 


=  y*J* g(a‘)  -  gC^jJdXCa1 


(II-35) 


The  Barankin  estimator  in  this  case  becomes 


T0(x)  =  J  Tr(a,x)dA(a)  +  g(aQ) 


(H-36) 
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Rrrthermore,  in  view  of  (11-32) ,  if  we  have  ah  infinite  sequence 
-{ gi(«)}  converging  to  g(a)  JfaeA  and  a  concomitant  sequence  {dA^} 

satisfying  (11-33)  and  such  that  for  some  dA,lim  S  /  7r(a,x)dA.  (a) 

**-  */  i 
A 
a 

o 

-  J' Tr(a,x)dX(a)|2  =  0,  then  (11-35)  ana  (11-36)  remain  valid. 

A  " 

a 

o 

While  it  is  admitted  that  in  many  cases  we  cannot  hope  to  find  a 

O 

closed  form  solution  for  dA  satisfying  (H-33),  two  facts  are  of 
interest. 

First,  as  has  been  mentioned,  the  use  of  any  admissible  A  will 

furnish  a  lower  bound  for  l*r  via  (11-11)  without  the  lUb  operation. 

In  fact,  to  this  end  we  need  not  restrict  A  to  be  a  signed  measure. 

Ihat  is,  if  dA  is  any  function  or  generalized  function  for  which 

//*( «  ,x)dA(a)<j)(x)dPa  (x)  =  J*/4>(x)-ir(a,x)dPa  (x)dA(a)  5 Jh(a)dX(a) , 

X  A  °  A  X  0  A 

then  (11-10)  still  holds.  In  particular,  if  h(a)  and  ir(a,x)  are  such 

that  dA(a)  =  6* (a  -  a  )da^  is  admissible,  then  with  a  taken  as  the 

o  o 

true  value  of  a  we  obtain  the  C-R  bound  for  unbiased  estimates  of  g(a) . 

^The  notation  6'(.)  denotes  the  derivative  of  the  Dirac  distribution. 
See  Reference  [37]. 
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Secondly,  If  the  parameter  space  is  not  conposed  of  a  countable 
set  of  points  one  could  perform  a  dissection  of  the  parameter  space 
into  say,  n  points  and  cast  the  problem  into  the  discrete  form,  men 
one  may  solve  for  the  suitable  constants  |aj  in  (11-22) .  By  taking 
n  large  enough  it  may  be  possible  to  obtain  a  good  approximation  to 
the  results  for  the  continuous  parameter  problem,  me  Barankin  esti¬ 
mator  which  results  from  this  procedure  will  be  unbiased  for  every 
point  in  the  dissection  if  a  solution  to  (11-22)  exists. 

In  arriving  at  the  Barankin  estimator  for  g(a)  it  has  been  noted 
that  in  general  T^x)  will  depend  explicitly  on  v  Such  an  estimator 
is  termed  "locally  best"  at  a  *  aQ.  If  only  a  locally  best  unbiased 
estimator  for  g(a)  exists,  it  is  not  legitimate  to  choose  aQ  to  cor¬ 
respond  to  the  true  value  of  a  for  the  purpose  of  constructing  the 
Barankin  estimator,  because  then  the  Barankin  estimator  will  explicitly 
depend  on  the  parameter  to  be  estimated.  In  this  case,  one  must  accept 
something  less  than  an  estimate  which  is  best  in  the  mean  square  sense 
about  the  true  parameter  value.  In  lieu  of  choosing  a  different 
approach  entirely,  it  might  be  satisfactory  to  use  the  Barankin  theory 
to  construct  an  estimate  which  is  best  in  the  mean  square  sense  about 
some  arbitrarily  chosen  point  aQ,  which  is  not  necessarily  the  true 
value  of  a.  (Implicitly  we  are  saying  that  the  selection  of  olq  is  made 
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frcm  data  which  is  statistically  independent-  of  the  data  to.  be  used  in 

constructing  the  Barankin  estimator.)  In  any  case,  we  will  obtain  an 

unbiased  estimate  of  g(a)  in  this  fashion  if  one  exists. 

We  might  then  hope  that  the  Barankin  estimator,  T  ,  which  is  best 

about  aQ  does  not  differ  widely  from  the  estimator,  Tp  which  would 

have  been  obtained  had  a  been  chosen  as  the  true  value  of  a.  As  a 

o  - 

measure  of  the  sensitivity  of  T  to  the  choice  of  a  we  could. use 

o  o 

„  ■Mv-V 

(ao  -  “l? 

where  the  expectation  is  taken  over  the  distribution  corresponding  to 
dp  the  true  value  of  a. 

Then, 


(II-37) 


.  Ehc  -  hi2  .  [s'%5  - 

^  o  *  * 


<“o  -  c‘l)' 


(ao  -  0l) 


(11-38) 


In  principle,  once  dXQ  and  dAp  (or  their  counterparts  for  the  discrete 
parameter  space)  are  found  satisfying 


f G(a,a’ |ao)dAo(at )  =  g(a)  -  g(aQ)  ;  YaeA 


(11-39) 


A 


y*G(a,a,|a1)dA1(a')  =  g(a)  -  g(a1)  ;  YasA 


(11-40) 


a  bound  for  S  can  be  calculated. 

‘This  concludes  the  present  discussion  c?  the  BarsSdLrs  theory, 
tfe  resark  that  the  extension  cf  the  store  to  the  treatment  of  rector 
parameter  estimation  prcoLems  is  irradiate  upon  replacing  a  Kith 
a  *  [a., ... ,a  land  allying  the  shore  notions  to  each  crrrxrent  c?  a. 


APrSSU  HI 

In  tMs  afceniix  we  present  the  actual  cat sMiter  prcggacs  used 
In  fee  slrulatlcn  of  Pfrp*yifligg  2  and  2. 

3k  first  listing  pertains  to  Eraqple  1,  Case  1-  To  cotain  the 
ctcer  cases  for  Seagate  1,  tae  values  of  SO  and  MO  (widen  carresrood 
to  SQ  and  oo)  were  dnanged. 

3k  second  listing  pertains  to  toe  sisulsticn  of  EtaqpSe  2a  Case 
1,  wSsere  only  t&e  center-i ieqaeacy  caysrianse  paraaeter  was  treated  as 
ssntoown. 

As  indicated.  ea  tee  note  Ifce*  tie  tfcSrd  listing  presented,  tnese 
statements  appropriately  sodU^  fcse  pregsras  for  Exanpla  2,  fese  I  in 
CEdsr  to  sfcalatg  Example  2,  Case  2. 


OU  OJOOUOUOOO  U  U  wl  O  DO  U  U  O  UU  DU  U  O  U  O  u  ■  J 


//.“Jliwv  ex£C  flBtlftiN 

SIKWUW  US  1  fit  lEiSI-Sa’i«iS  ESIIMIE 

IMPLICIT  REAL**  *M» 
hI=WJ.  Or  «KIE-C1KIH  TRIALS 
K2=M4S.  IS-  U«e  SAMPLES 
T* JBSER^aT  IJ?T  IEWJTh 

REGRESSION  PARAMETER 
BO=l3MlliO««  CB/iRIJUtCE  PARAMETER 
G1=VECTIX  Qf  K  K0N»  REGRESSION  FWi'TIUM  VALUES 
RzD&aA  VECTOR 

U=  VECTOR  t!F  IMOEP.  GAUSS.  R.VS 
E=VECTCPt  CP  CORRELATED  GAUSS.  R.V.'S 

VRIIcl6.IOI 

lO  rCJSMAT I lHl  ,  *  SIMULATION  FUR  L.S.  ESTIMATOR  tr  &LPH4*//I 
THE  PROGRAM  HILL  BE  LIMITED  TO  A  max  tr  TEw  TInE  SAMPLES 


ECWH  KRAJKDG 

DI)0G!(n  UIIOI.EIIOI.GKIOI.RUOI.PHIOUO.IOI 
DIMENSION  RHCOUTIDw  AMAT1  (II*  AHAT2I1I 
KJUJLE  PRECISION  T,aO,RS«a,B 

INPUT  iKErsiUlICSt 
Nl=1000 
Ss2=20 
1=1.0 
bq=-oi 

A 0=2.0 

KMA0G=1 

iasD*!!*!!!!!!! 


SCKE  AiSXILLIARY  ClittSTAMTS 

?i*3.14IS926 

H0=1QC»PI 

FC=H0/I2*PI1 

BELTA=T/ IH2-1I 

WRITEI6,  15III1  fM2,T,BQ,A0,F0 

IS  FORMAT ( IhO, *  Hl  =  * ( I6« *  M2=* » I4»  *  T=*,FS.l,*  B0=',F4.2. 

I*  A0=*,F4.2,«  F0=*  ,F8.5, *  CPS*//I 

FORMING  The  VECTOR  Gl 

00  GO  K=1*N2 

L=K-1 

40  G1IKI=COSIKO*DELTA*L1 

PRE— CUMPUTAT IONS  FUR  L.S.  ESTIMATE  OF  AO 
Y1=0.0 
OO  5S  1=1, M2 
5S  Y1=G1IIJ**2*Y1 
TI=YI 

THE  VARIAHCE  OF  AHATI  MAY  BE  CALCULATED  ANALYTICALLY  AMD 
HILL  BE  COMPARED  WITH  THE  SIMULATION  RESULTS 

FIRST  HE  CALCULATE  THE  HlNlKUH  VARIANCE  FUR 
ESTIMATiuN  OF  AO — VAHIK 


on  no  non  non  on 
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PHU*yeXP*-ac»2ELlAl 
2  =  IG1  111**21*11. -RHD»*2I 
CO  fry  1 =2,K2 

60  Z=tG!(Il-RM»*Glil-ill**2-*2 
V4Wfpll.-R«!*»2l/2 

SETTING  UP  THc  TRUE  CUV.  MATRIX, PHJO 
UO  199  1=1 ,«2 
DO  100  J=  I  ,1«2 
PH10I I,Jl=RHffi**U-Il 
PM1G1J.1I=PHI3II,J1 
lOO  CllRTIMit 

CACCUtAl IM&  iss  VARIANCE  Ur  TmE  L-S-  ESTIMAlfc 

k=0.U 

DO  90  1*1, M2 
CO  90  J=1,W2 

K=G1II1*PHI9U,J1*S11J1-*W 

90  CONTINUE 

YLS*W/ 111**21 

MniE-CuiC  SlKUUTltVi 

B=USS«T  1 1  ,-RHL“*2l 
X1=0.0 
X2=0 .9 
xa*o.o 
Y7=0.0 
YS-O.Q 
Y5-U.C 
Y10=0.C 
Y1 1=0.0 
!(=l 

DO  509  Hb=  1.N1 

GENERATING  INSJEP.  GAUSS.  R.V.'S 
DO  20  L=i,10 
UlLl=RAIii)Gl  1RAND1 
20  CONTINUE 

GENERATING  The  OESIRED  SAMPLED  RANDOM  PROCESS 
Em=um 
DO  30  K=2.10 
EIK?=B*U(Kl+RHG«ciii-I  ! 

30  CONTINUE 

C  GENERATING  THE  DATA  VECTOR 

DU  SO  1=1, M2 
R«n=AO«GlC  I1+E1II 
50  CONTINUE 
C 

C  THE  t.S.  ESTIMATE - AHAT1 

Y2=0.0 
DO  65  1=1, HZ 
Y2=G1 (IJ*Rf II+Y2 
65  CONTINUE 

AHAT1(n;=Y2/11 


C  HllMliWi  Mr  RHO 

Y3=0.t3 

m  70  i=z,m2 

T3=IRI  1  J-4H171!MI»il  (  I  )  1*IR{  t-i  J-iHiH  CM«t-!  I  !-l  » J+Y3 
70  Cir*T!l«;fc 
Y%=0.0 
IH=»s2-l 
00  75  1=1. in 

Y«=  I K 1 1 1 -AHiT  llfl  I  •&  I  C  III  «*2  *Y<. 

75  CVMlUOSt 

RHOKAT 1K1=Y3/Y* 

€ 

C  7 Hi  KEV15EU  ESTIMATE - AHAT2 

Y5=Kill*{»llll*f  i.-KHJriATIMl»»21 

Mi  200  1=2, M2 

200  Y5=«KI  1  l-RHIBtAT  IHl*RfI— 1|1»IG1II  J»Yi 

Xm 

y&= i si i n **2 i * e i .-RmiHAT : m j * =2 i 

IKi  210  1=2, K2 

210  Y6=(SM I1-RHUHATIM»=G1II-1I»**2»Y6 
AKaT2  (III  =V5/Y6 

c 

C  Kfc-fcSsIHAlltW  IBF  RWi! 

X*=0.0 

Ml  220  I=2,tt? 

X*=IRII.»-AKA  21H J»G1  IIJ1»(KI1— 1 1-AHAT2IW  J®{»1  S  1  — 5.  5  1-rX* 
220  CtAllK)!: 

*5=0.0 

1HJ  230  1=1,  I* 

*5=1 Kl 1 I-aKa721*J«GI ( I  J1**2*X5 
230  CUMTlMib 

KHUKAT I  Mi  *XVX5 
C 

C  THE  1TEK*7££>  ESTIMATOR  OF  ALPHA - AHAT11 

X6=Ki  1.1*011  S  1  =  5  J  RHUHA1  1 

00  2*0  1=2. K> 

2*0  Xt='l<el  |  J'-RJr>l!«>A7 1—111*10111  J— KRltMAT  IM)=G1  ( I— 1 1  )-»X6 
X7={SiUl**Z?*l  i.-KMtxHATlwi*=2i 
Ml  2*0  :=2,f»2 

260  S?=IG1 1 1  l-HI»UHAT|fll=Gl  11-1 1  J=*2*X7 
AHA71T=Xt,/X? 

C 

C  SAHrLc 

Y7=Y7*KKy«AT  t»l 
7t5=Y8-»A*tA71I«l 
Y9=Y9»AHftT2101 
C  hfcA*v-SlRJA#>.<:0  ERKUK 

Sl  =  lAHATlffli-AO»**2+Xl 
vh7=CAHAT2sM)— A01==2*X2 
a3=£SH0HAT (NJ-RHU)*=2+X3 
C  RSAH  iW  KSfc  Or  ITERATED  tSTIMATk 

YlG=Y10+AHATIT 
Yl5=Yil+ (AHA 7 IT-AO) e=2 
500  CU;«TIP<Ut 
HKHO=Y7/«l 
HAI=Y8/K1 
t*.a2=y9/Hi 

c 

<H5tAl=Xl/Ml 

KSfcA2=X2/Nl 
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MSERH0=X3/hl 

c 

KilT=rlO/Nl 

hSfcalT=Yll/«l 

pEkF=vakih/ksEaII 

c 

MRITE(6,7) 

7  FORMAT (1h0.(  AO  KA1  HA2  VLS  KSkAl  HSEA2* 

1»  VAMIN  KRKO  KSERHU  KAIT  KSEAlT  PERF*,//I 

c 

WRITE(6,I2»A0,KA1,HA2,VLS.MSEA1,HSEA2.VAHIM,KXHU,KSEKH0. 

1HAIT .KSEAlT ,PERF 

12  FORKATllH  ,FA.1,2X.2(F6.A,2XJ,F6.3,2X,2(F7.4,2X1,F7.4,3X, 
1F6.*,2X,F8.A.*X,F6.«,2X,F6.A,4X,F8.61 
WRITE(6,1*)IRAKD 
If.  FORMAT (1H0>*  IRAKO=  • ,1101 
EMI 

C  SUttKOlT IhE  FUR  GENERATING  GAUSS 1 Ah  RAN1XJR  VARIABLES 

FW.CTIU’*  RAMOGI  IRAKLI] 

Of  TO  (1.21,  KRAhDG 

1  Ul=RAJ»SW*.l  IKAhlll 
U2=RAM8JKt  IKAfOI 

XKAG— SbRl {-2.»ALU'J(U111 
XI =XKAG»COSI 6.283 1 8f  »U2 1 
KSKUGsX! 

&RAM1G-2 
GO  TO  20 

2  X2=XHAG*SI«(6.28318A*U21 
RAk0G=X2 

KRAffl)G=  1 
20  RETURII 
EhU 

/» 

/£  EMI  OF  JU3 


JUUUOwHII/OOOwlull.KIU  <J  O  O  O  O  U  O  o 


no 


//MSM44  EXEC  FORlRAHiBCG? 
implicit  real»4  ipo 

CALS.  CLUCK CXKKI 
KKN*XKR 

SIMULATING  7 HE  cask  FOR  HcTEp.UOYmEO  NUISL 

Ul-KiK  UP  HUJ*7E~CA?.Lii  TRIALS 
H2=HU.  OP  TIME  Si--XPi.ES 
T=«JSE»VaT!BH  LENGTH 
ASnUKXKUiW  REGRESS JUN  PARAMETER 
CCoftOiSt  BANOVIOTh  PARAMETER 
«>Stt?!5E  CENTER  FRfcSUENCT  PARAMETER 
W<SE5SESJU1»  rU^CT  UiH  FXciKJENCY  fKHUW.NI 
Ui=KHUWK  REGRESSION  YECTUR 
H*»JA7A  YtCTUS 

0=VcCt3J  JM-  IH05P.  GAUSS,  S.V.*5 
V»*CXILIASV  VECTOR.  Or  IftsJEP,  GAUSS.  R.«.*S 
£*V££70R  fir  CORRELATES*  GAUSS.  R.V.'S 


CUMtfN  KKAKOS 

uihehsiu«  un.os.ttioi ,c-iuoj .Riios,v;2) 

D1KE.NSHJM  PHlGi  lOj 101  rPU jHAT  21.1!j105.L111GI*L2I  10) 

DOUBLE  PRECISION  FHIOil’Hl HAT  »  BO.lfG  lAAOiBBO.CC  »HO»Hl  ,C3*C4« 
•C0,C1,C2,CS 
DOUBLE  PRECISION  RHU 
INPUT  INFORMATION 
Hl=10C"0 
S*2=1G 
T=1.0 

bo=.oi 
*0»2-0 
H0=3.0 
KRAM>G=1 
17  =  1 

iKAfto=nnmm 

P!*3.1415*>26 
HC*3*PI 
FC=*C/ l Z*P II 
FO*W&/'Z*PI> 

0ELTA»T/I.N2-11 

MR!TES6,15)Kl»H2,r,H0,A0«F0,FC 

15  FOKKATllKO, ‘  Nl=* ,I6T»  N2=s,|4,«  7=*,F5.1,*  B0=*,Fa.2, 

1*  AO3 1 »F4 .2 f 1  F0“‘ «F6.3t *  CPS  ?C=’,F6«3,‘  CPS*//) 

FORMING  THE  VECTOR  Gi 

00  AO  K<=1  .N2 

L=K-I 

40  GltK)=COSIWO»OcLTA*U 

PRE-COMPUTATIONS  FOR  L.S.  ESTIMATE 
Y1=0.0 

OU  55  1=1 ,H2 
55  Yl»tlf J)*»2+Y1 
Tl  =  Yl 

setting  up  the  TRUE  COVARIANCE  MATRIX 
OU  100  1=1 »N2 
00  100  J«1,N2 


no  o  or  or  oooo  or*  noon 
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PH 10(1 , JI=UEXP(-bO»UELTA»( J- I J )»OCUSlHC*UfcLTA»l J-I J ) 

PHIOt J , I 1=PH!0( 1 , J) 

100  CONTINUE 

c 

C  CALCULATING  THfc  VARIANCE  UP  THfc  L.S.  tSTIHATt 

H=0.0 

LO  SO  1=1, N2 
UU  90  J=1,N2 

W=blin*PH10II.Jl*Gl(J)«H 

90  con  •  i wit 

VLS=K/(T1**2» 

c 

C  CALCULATING  REGUlREU  CONSTANTS  TU  GENERATE  THE  OESIKEU 

C  TIKE  SERIES 

C 

AAU=PHlOl 1,21  =  11 .-UEXP  t— 2*B0=DEL I A ) ) 
bBO=l.-OtXP(-A»HO«UELTA) 

CC= IBBO+USURT {BB0=o2-A»AA0**2 J 1/2. 

CO=USOKT ( 1-CC»*2 ) 

H0=llSORT  (CC) 

Cb=H0»AAU/CC 

CA=0EXP1-2«H0«=UELTA} 

C5=2»PHIO(1.2I 

Hl=H0»C5-C3 

Cl =( PHI oil ,2)-H0=Hl l/CO 
C2=1.-(H0=»2+H1=»2I-C1»»2 
C2=USOKTlC2 I 

FUK  THE  kcST  OF  THE  PROGRAM  HE  NEtD  THE  MATRIX  INVERSE 
UF  PHIO — NUT  PHIO. 

THfc  FOLLOWING  MATRIX  INVfcRSION  KfcPLACES  PHlO  KITH  ITS  INVtKSE * 

CALL  DPlNVlPHlO,N2,TEST,UfcTER,Ll,J.2,N2l 

Wfc  NOW  CALCULATE  THfc  MINIMUM  PUSSIbLfc  VARIANCE 
FUR  ESTIMATING  AO - VAH1N 


HI =0.0 

00  95  1=1 ,N2 
UU  95  J=1,N2 

W1  =  &UI)*PH10(  I,J)*G1(J)+H1 
95  CONTINUE 
VAM I N= l • /HI 


MONTE  CARLO  SIMULATION 

X1=0.0 

X2=0.0 

X3=0.0 

Y7=0.0 

Y8=0.0 

Y9=0.0 


00  500  N5=l ,N1 

GENERATING  THE  SfcTS  UF  INUEP.  GAUSS.  K.V.'S 
i)U  20  L  =  1 , 10 
U(L)=RANUG( IRANU) 


ovi  u  o  oo  uuuoo  oa  oo 
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20  CUNT INUfc 

UU  21  L=l,2 
V(L)=RANUG(  IRANI)) 

21  CUNT INUfc 

GfcNfcKAT  ING  THE  OfcSlKEl)  SAHPLfcO  R.P. 

Ell )=HO»UC l ) ♦CO*V( 1 ) 

E  C2 ) =HU=Ul2 )+Hl=UI 1)+C1=V(1)*C2*VI2) 

UU  30  K=3,N2 

fc  (K ) =C5*fc IK-1 )-C5*fcU-2)*H0»UIK)-C3*UIK-l > 

3U  CUNI INUfc 

GENERATING  THE  DATA  VECTOR 
1M1  30  1=1,N2 
*tm=Ao»Gim+Eii) 

50  continue 

THc  t.S.  EST IMATfc - AHAT1 

Y2=U.O 

on  -55  I  =  1,N2 
Y2=Gim»RIl)*Y2 
65  CONTIMJfc 
AHAT1= Y2/ 1 1 

fcSTIHAT  IlfN  OF  THE  COVARIANCE  MATRIX 

THIS  PKOGEAH  TRfcATS  BANUKIUTH  AS  A  KNUWN  PARAMETER 
HE  fcST JflATfc  PHlO (  1  >2 )  AND  THfcN  THETA=WC*UfcLTA 
AHAT=AHAT1 
73  Y3=0.0 

011  70  !=2fN2 

Y3= IRI I  )-AHAT“61  CIIIMRII-l  )-AHAT=GlI  1-1 )  )+Y3 
70  CONTINUE 
Y4=0.0 
IM=H2-i 
OU  75  1  =  1,  IN 

Y4=?KI  !  »-AhAT*GU  i  ))**2*«f 
75  CUNTPAit 

?HIH£Tii,2)=Y3/V<r 

THL TA=AKl, f,S  1  SNGi  I  PH IHAT  11,2))  =5Ntt  i Db.xP I  30*1*11  .=  ?  ?  ? 

SETTING  UP  THE  ESTIMATED  COVARIANCE  MATRIX 
DO  300  1=1, N2 
UU  300  .»-I,N2 

PHlHAT<l,J)*OEXP(-By»OfcI.TA»U-I  )  )>COSlTHfcT&slO-l  / ) 
ph;hat(j,p=phshat i  i,j> 

300  CONTINUE 

IHfc  REVISED  ESTIMATE - An£T2  UK  AHAT1T 

CAlC  DPINVIPHIKAT ,NZ ,TtST .DETER, L 1 ,L2,N2 ) 

HZ'**).)) 

OU  310  I«l,N2 
DU  310  U=1,N2 

W2=fcl f i)=?HIHAT(l tJ)«GlI j)+«2 
310  CONTINUE 
W3*0.0 

OU  315  3=1, N2 
OU  315  J=1»N2 

W3=GH  1  l<-?M!rWn  1  ,J)*R(  J3  *»3 
313  GONT INUfc 
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AHAT*W3/H2 
1F11T-1)  63,63,64 

63  AHAT2=AHAT 
1T«2 

GU  TO  73 

64  AHAT1T*AHAT 
1  T  =  1 

C 

C  PERFORMANCE  CALCULATIONS 

C 

C  SAMPLE  MEANS 

Y7=Y7+AHAT1 
Y8=Y8+AHAT2 
Y9= Y9+AHAT 1 T 
C 

C  HEAN-SQUAREU  ERROR 

Xl=(AHA71-AO)**2+Xl 
X2* ( AHAT2-A01 **2+X2 
X3=CAHATIT-AO)**2+X3 
CALL  CLOCKIKLL ) 

XKM=KLL 

I F( 1 A8S IKKH-KKN) .GE .  252001  GU  TO  1500 
500  CONTINUE 
1500  CONTINUE 
HAl=Y7/N5 
HA2=Y8/N6 
HAIT=Y9/n5 
MSEA1=X1/N5 
KSfcA2=X2/N5 
HSEA I T=X3/N6 

perf=vahin/mseait 

WRITE ( 6, 7 ) 

7  FORMAT :iHO,‘  AO  MAI  HA2  VLS  MSEA1  HSEA2' 
1«  VAhIN  HA  IT  MSEAIT  PERF',//) 

C 

mRITE‘6,12 l AO ,HA1 ,MA2 »VLS ,HSEA1 ,HSEA2, VAHIN, MAI T .MSEAIT  » 
1PERF 

12  FORMAT ( 1H  , F4.1,2X,2(F6.4,2X),F6»3,2X,2(F7.4,2XJ,F7.4,4X, 
IF6.4,2X,F6.4,4X,F8,6J 
V#RJT£(6, 11001  N5 

1100  FORMAT ( iH  ,«  ACTUAL  NO.  OF  ITERATIONS^  *,18/) 

WRITE (6, 14)  IRANO 
14  FUrtiiAT  J 1H0,  *  IRrtNO=  ',110) 

END 

/» 

/C  ENO  OF  J’jB 


j 


non 


llH 

/ /kain^a  txfcc  Fortran 

Hfckfc  BANDWIDTH  AND  CEN I ER-FREOUENCY  AKt  REGARDED  AS  UNKNUHN 
Wfc  fcSTIHATfc  PHIOl 1,2)  AND  PHIOtl,3>  AND  THEN 
THETA=WC»OELTA  ANl)  EXP  1 -BO*DEL T A ) =RHU 
AHAT=AHAU 
73  Y3=0»0 
Y4*0.0 
Yi>=0.0 

DO  70  1=2, N2 

Y3=1R1I )-AHAT$Gl I  I ))*(KI 1-1 )-AHAT«Gl 11-1) )  +  Y3 
YS=  IRC  I )  -AHAl  of.l  1  1  )  )=  =  2+Y!> 

70  CONTINUE 
DU  71  1=3, N2 

Y4=(K<  1  )-AHAT=Gl  1  1  )  )  *  1  KC1  -2 ) -AHAl  *G1  1  1-2D+Y4 

71  CONTINUE 
PH  I HAT ll,2)=Y3/Y3 

PHI  HAT  11  ,3)  =  Y4/lYS-lR(2)-AHATwG112n«’»2) 

THETA=ATAN1 SNGt IDSOKT IDABSll.-PHlHAT 1 1,3)/PH1HAT 1 1 , 2  )**2 ) ) ) ) 
RHO=PHIHAT  11,2)  /ClIS  l  THETA  1 
C 

C  SETTING  UP  THE  ESTIMATED  COVARIANCE  MATRIX 

DU  300  1=1, N2 
DU  300  J  =  1  ,N2 

PHIHATI I,J)=IRHU**1J-I ) )=CUS1THETA*1 J-I ) ) 

PHI  HAT  1 J , I ) =PH IHAT { 1 , J ) 

300  CONTINUE 

/« 

/£  END  OF  JOB 


‘These  statements  replace  statements  73  through  300  in  the 
previous  listing  when  it  is  desired  to  simulate  Case  2  of 
Example  2. 
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number  of  problems  that  arise  in  radar  and  sonar  applications  can  be  regarded 
as  parameter  estimation  problems,  in  which  the  desired  signal,  f(t,a<),  is  imbedded 
in  non-white,  Gaussian  noise.  It  is  desired  to  estimate  the  unknown,  nonrandom 
,  parameter  vector,  a,  from  observations  (continuous  or  sampled)  of  the  received  noisy 
I  signal  over  a  finite  time  interval  [0,  T],  Here  f(t,  a)  is  a  known  nonstochastic  function, 
and  we  shall  consider  the  case  when  f(t,a)  is  linear  in  a.  In  this  case,  a  is  referred 
to  as  a  linear  regression  vector  "(U)*'  .  ■ 

We  shall  investigate  the  variance  the  Least-Square  (LS)  estimator  and  of  the 
so-called  Generalized-Least-Squares  (GLS)  estimator  for  a.  Both  are  unbiased 
estimators  for  a. /  ’,JU)7T"' 

When  the  noise  covariance  function  is  completely  known  one  may  construct  a 
minimum  variance  unbiased  estimator  (MVUE)  for  a,  and  this  estimator  is  a  member, 
of  the  class  of  GLS  estimators.  "(U)" 

Our  interest  is  in  the  case  when  the  noise  covariance  is  not  completely  known, 
but  may  be  regarded  as  a  known  function  of  a  finite  number  of  unknown,  nonrandom 
parameters,  £.  "(U)" 

It  is  shown  that  when  £  contains  any  covariance  parameters  other  than  the  noise 
variance,  there  exists  no  MVUE  for  a.  "(U") 

However,  we  shall  exhibit  a  class  of  problems  for  which  the  MVUE  for  a  has  a 
variance  which  is  orders  of  magnitude  smaller  than  that  of  the  LS  estimator.  In  such 
a  case  it  ia  of  interest  to  find  an  estimator  which  makes  use  of  whatever  covariance. 
information  is  available  in  an  attempt  to  approach  the  performance  of  the  MVUE.  "(U)'1 
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Ttem  13  Abstract  (con’t) 


It  is  shown  that  we  can  significantly  improve  upon  the  LS  estimator  by  employing 
a  bootstrapping  procedure  to  estimate  o.  In  some  cases  the  bootstrapped  estimate 
of  a  can  be  shown  to  be  unbiased.  In  any  case,  it  is  demonstrated  via  computer 
simulation  that  the  bootstrapped  estimate  of  a  is  capable  of  reducing  the  variance 
of  the  LS  estimate  by  orders  of  magnitude.  In  fact,  the  mean  squared  estimation 
error  using  the  bootstrapped  estimator  for  a  may  be  within  a  few  percent  of  the 
variance  of  the  MVUE,  i.  e.  ,  the  variance  tKe  MVUE  would  have  if  fi_  were  known 
a-priori.  "(U)" 

The  bootstrapping  procedure  consists  of  using  the  LS  estimate  of  0,  to  provide 
an  initial  estimate  of  the  regression  vector  from  which  an  initial  estimate  of  the 
unknown  covariance  parameters  is  constructed.  "(U)" 

Two  procedures  are  outlined  to  accomplish  the  estimation  of  8_.  The  first 
approach  is  based  upon  an  application  of  the  theory  of  locally  best  unbiased  estimation 
The  second  approach  is  herein  termed  the  "inverse-covariance-fuhction!'  technique. 
Because  of  its  simplicity,  the  latter  approach  is  employed  in  the  simulations.  "(U)" 
Regardless  of  the  manner  in  which  the  covariance  parameters  are  estimated, 
these  estimates  are  used  to  construct  the  GLS  estimator  for  a.  This  is  the  first 
iteration  of  the  bootstrapping  procedure.  "(U)" 

The  GLF  estimate  of  a  is  then  used  to  re-estimate  the  unknown  covariance 
parameters,  and  then  to  re-estimate  the  regression  parameters.  "(U)" 

The  process  uses  only  the  one  available  record  of  data,  and  may  be  repeated 
ad  nauseam.  However,  dramatic  results  were  obtained  after  only  two  iterations 
of  the  bootstrapping  procedure.  "(U)" 
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